Compatibility-based source-pair screening for language model merging: An empirical evaluation

Abstract

Relevance. Expanding repositories of specialized language model adapters make exhaustive source-pair evaluation costly. Purpose. This study evaluates whether pre-merge compatibility descriptors provide predictive and selection value beyond domain composition and direct feature ranking. Methods. Twelve low-rank adapters sharing a pretrained backbone yield sixty-six conversational and mathematical candidate pairs. Task-vector direction, sign interference and centered kernel alignment predict merged-model benchmark scores. Evaluation includes training-only cross-validation, sixteen held-out pairs, feature ablation, same-domain prediction and adapter-, configuration- and recipe-disjoint holdouts. Results. Linear regression achieves held-out correlations of zero point eight five eight for conversation and zero point eight zero one for mathematics. However, a pair-type baseline reduces mean absolute error from zero point zero two four eight three to zero point zero zero seven eight one for conversation, and from zero point zero one nine zero eight to zero point zero one zero zero eight for mathematics. Adding compatibility to pair type modestly improves mathematical error but worsens conversational error. Both the predictor and representation similarity recover the best held-out pair with a single selection. Grouped results reveal unstable calibration and no consistent selection advantage; same-domain correlations are negative. Conclusions. Domain composition explains much of aggregate predictability in this repository. Compatibility-based screening has repository-specific ranking value, but supervised calibration does not establish superiority over simpler baselines. Cost reductions remain conditional projections because successful evaluation-runtime measurements are unavailable. Candidate growth remains quadratic, and calibration, caching and transfer determine practical savings.