CrossMean: Model Merging by Pairing Merged and Expert Activations

arXiv:2610.14762 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CrossMean: Model Merging by Pairing Merged and Expert Activations".

Jane: The gist The authors introduce CrossMean,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up this paper, "CrossMean: Model Merging by Pairing Merged and Expert Activations," what’s our big picture here? It seems like the authors are pushing a specific way to handle model merging that goes beyond simple summation or averaging of expert weights.

Jane: They are proposing a method where you explicitly pair the activations from the merged layers with those from the individual experts using this bilinear statistic, which they call CrossMean.

Lu: This approach is important because it shows that just using one side of the equation, like a standard Gram matrix on expert inputs or even just re-measuring statistics in a merged model, isn't enough to capture all the necessary information for high-quality merging.

Meng: For practical AI engineering at scale, this means that if you are trying to merge models without retraining, you might benefit from using this paired statistic because it gives you a better performance indicator on specific task sets.

Tom: Exactly. They show empirical results where CrossMean improves over RegMean and other methods across five vision backbones and BERT encoders on eight different task sets, which is a solid comparison of how much better it is under the right conditions.

Jane: The findings are that this pairing strategy isn't universally better for every single merging scenario, but it’s definitely beneficial when you consider where the merged-prefix mismatch between the structures is large.

Lu: And they quantify this mismatch using that label-free alignment criterion ρ, which tells you exactly how much benefit CrossMean gets over other methods based on that geometric relationship between the two statistics.

Meng: So it boils down to this: if you want better merging performance, calculate that alignment criterion first and find the task sets where it's low, and then use CrossMean there.

Tom: That’s the practical implication—it guides your experimental design. The paper demonstrates that CrossMean provides a measurable advantage over existing closed-form methods when you account for this specific pairing of merged and expert activations.

Conclusion: Tom: So we've seen how they built CrossMean, which pairs merged activations with expert ones to improve model merging methods by using a specific bilinear statistic instead of just standard statistics on one side.

Jane: It’s basically taking what you get from the merged network and comparing it directly to what each individual expert contributed, which is a really smart way to look at things.

Lu: The authors showed that they formulated this as a linear system, and they proved that their new statistic actually uses the same matrix on both sides of that equation, which is interesting because most people just use one side for those calculations.

Meng: From an engineering standpoint, what does this mean for us shipping models? Does it add too much overhead or complexity when we are trying to merge things quickly?

Tom: It doesn't add a huge amount of overhead; the cost analysis shows CrossMean actually needs less peak memory than some of the other methods we looked at.

Jane: And they found that this method performs better on eight different task sets across five vision backbones and BERT encoders, which shows it’s not just theoretical stuff.

Lu: The label-free alignment criterion they derived is a way to measure how much benefit you get from using this pairing, and it tells us exactly when CrossMean outperforms other approaches.

Tom: So the main point is that you don't just merge things blindly; you check where the overlap between the merged parts and the experts is most useful for your specific task.

Jane: It suggests that for real-world merging, we should look at these alignment criteria before we even start training or merging.

Lu: And they’ve also shown this works even when dealing with deeper layers, which is a good sign for scaling things up to very complex architectures.

Meng: So if I'm looking at deployment, I need to know that this method scales and doesn't introduce new bottlenecks on the actual hardware.

Tom: It seems they’ve done a pretty solid job on the mechanics and showed it delivers better results under those specific pairing conditions.

Moshe Eliasof

Ben-Gurion University of the Negev

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The authors introduce CrossMean, a model merging method that pairs merged and expert activations to improve upon existing closed-form methods by replacing standard statistics with a bilinear

Key concepts

Model Merging: This technique combines multiple fine-tuned models into one single network without retraining. The authors focus on how to calculate the optimal weights for this merged model efficiently using statistics derived from the original experts.
Closed-Form Methods: These are mathematical formulas used to compute layer parameters in a merged model directly from input activation statistics, rather than through iterative training. The paper analyzes these methods by showing they all rely on the same underlying matrix structure.
CrossMean Statistic: This is a new bilinear statistic that pairs the activations of the merged network and individual experts. It replaces standard linear statistics by calculating a cross-statistic along a path from the merged prefix to an expert, allowing for better alignment between different model states.

Terminology

Model Merging — 1× in this paper · explained in 2 episodes

Summary

The gist The authors introduce CrossMean, a model merging method that pairs merged and expert activations to improve upon existing closed-form methods by replacing standard statistics with a bilinear statistic

How it works

The paper formulates existing closed-form merging methods as a linear system per layer, showing that they all use the same matrix on both sides, which is typically a Gram matrix measured on a single network <ref:2610.14762#pg3>. However, CrossMean introduces a bilinear statistic that pairs merged and expert activations to replace this matrix <ref:2610.14762#pg3>. This new statistic is defined in Equation (4) as C¯t = X˜⊤t 1/S Σ s∈S Xt(s), which averages the cross statistic along a path from the merged prefix to the expert <ref:2610.14762#pg5>.

The core of CrossMean involves replacing the Gram matrix on the right-hand side of Equation (1) with this bilinear statistic, leading to a new linear system where Bt = C¯t <ref:2610.14762#pg5>. This replaces Progressive RegMean's use of the Gram matrix X˜⊤t X˜t with a cross statistic that pairs merged and expert inputs <ref:2610.14762#pg5>. The set S = 0, 1/2, 1 is used in experiments to include both the Gram matrix of the merged inputs and the cross statistic itself <ref:2610.14762#pg5>.

Key Contributions

The contributions of this work are summarized as follows:

• We extend the linear-system view of closed-form merging, discussed in Section 2, to a broad set of existing methods, and we show that all of them use the same matrix on both sides, which for the activation-based methods is a Gram matrix measured on a single network <ref:2610.14762#pg3>.

• We introduce CrossMean, a closed-form merging method whose bilinear statistic pairs the merged and expert inputs, and which contains RegMean and Progressive RegMean as special cases <ref:2610.14762#pg5>.

• We propose a label-free alignment criterion, computed before the closed-form merge, which is lower where Progressive RegMean and CrossMean improve more on RegMean <ref:2610.14762#pg7>.

• We show empirically that CrossMean improves on RegMean on the eight-task sets of five vision backbones and on BERT encoders, and that with all linear layers merged, it further improves on the depth-ordered correction on each backbone that we tested <ref:2610.14762#pg6>.

Alignment Criterion

A label-free criterion is derived to indicate how much a merge accounts for the merged inputs, defined by comparing two statistics involving the merged inputs: A = X˜⊤t X˜t, C = X˜⊤t Xt <ref:2610.14762#pg6>. The alignment criterion ρ is measured as the cosine of the angle between these two statistics in the Frobenius inner product, ρ = ⟨A, C⟩∥A∥F∥C∥F ∈ [-1, 1] <ref:2610.14762#pg7>. A lower value of ρ indicates a larger angle between the two statistics and suggests that CrossMean and Progressive RegMean improve more on RegMean <ref:2610.14762#pg3>.

Empirical Results

Empirically, CrossMean improves on RegMean on all five backbones, and achieves a higher accuracy than all other methods in eleven of the fifteen settings of the shared-protocol grid <ref:2610.14762#pg8>. Furthermore, CrossMean improves further with full coverage and when applied to LOT <ref:2610.14762#pg6>. The alignment criterion ρ anticipates this variation, showing that CrossMean improves more on the task set with the lower ρ <ref:2610.14762#pg5>. The asymmetry of the bilinear statistic is important because removing its antisymmetric part lowers the accuracy of CrossMean monotonically on both backbones that were tested <ref:2610.14762#pg2>

Cost and Calibration Set Size

The cost analysis shows that CrossMean and Progressive RegMean need less peak memory than RegMean <ref:2610.14762#pg16>. Specifically, on one NVIDIA B200, CrossMean takes 31.2× the time of RegMean on the eight-task set of ViT-B/32 <ref:2610.14762#pg16>. Furthermore, CrossMean and Progressive RegMean need 28 to 34% less peak memory than RegMean because they do not hold the Gram matrices of all experts at once <ref:2610.14762#pg16>. The plateau for CrossMean is reached at about four times as many images compared to RegMean and Progressive RegMean, suggesting that the 256 calibration images per task used in vision experiments lie on the plateau of CrossMean <ref:2610.14762#pg17>.

Conclusion

Overall, our results suggest that merging benefits from pairing merged and expert activations <ref:2610.14762#pg2>. This is demonstrated by CrossMean's superior performance over RegMean on eight-task sets of five vision backbones and BERT encoders, and its sensitivity to the merged-prefix mismatch quantified by the alignment criterion <ref:2610.14762#pg2>. The findings indicate that the bilinear statistic is beneficial in particular where the merged-prefix mismatch is large <ref:2610.14762#pg5>. The results hold across backbones and numbers of tasks, suggesting a general benefit to this pairing strategy <ref:2610.14762#pg2>. The study also confirms that CrossMean improves on LOT merging by +3.75% to +2.11% depending on the task set's alignment criterion <ref:2610.14762#pg19>. The results suggest that merging benefits from pairing merged and expert activations <ref:2610.14762#pg2>.

Improvements for AI systems

Bold header: CrossMean Merging Mechanism

The system can merge fine-tuned experts into a single network by using a bilinear statistic that pairs merged and expert activations, which replaces the Gram matrix used in existing methods, leading to an improvement on merging accuracy, as shown by the empirical result that CrossMean improves on RegMean on all five backbones.

Bold header: Label-Free Alignment Criterion

The system can derive a label-free criterion that indicates the task sets in which accounting for the merged activations improves the merge more, which is computed from statistics of both input distributions, allowing researchers to select optimal merging strategies before performing the actual merge.

Bold header: Depth-Order Correction Integration

The system can effectively address merged-prefix mismatch by implementing a depth-ordered correction, where it re-measures inputs on the merged prefix, which allows for superior performance compared to methods that only measure statistics on single experts.

Bold header: Task Set Optimization

The system can achieve superior merging gains by selecting task sets with a lower value of our criterion [ρ], as this criterion is defined as the average of this cosine over the mergeable layers and the experts, indicating which task sets benefit most from pairing merged and expert activations.

Bold header: Computational Efficiency Gains

The system can reduce computational overhead during merging by requiring less peak memory than RegMean and showing that CrossMean takes 31.2× the time of RegMean on ViT-B/32, suggesting a more efficient implementation for large models and many tasks.

Bold header: Robustness to Activation Changes

The system's performance is robust to activation functions; specifically, switching from GELU (G) to QuickGELU (Q) only changes the criterion by at most 0.004, indicating that the core benefit of pairing activations remains stable regardless of the specific activation used by the experts.

Bold header: Transferability to Different Architectures

The system can be applied effectively beyond vision encoders, as demonstrated by its success on BERT encoders and GLUE benchmarks, suggesting that the bilinear statistic is beneficial beyond image encoders.