Dynamic Model Merging Made Slim

The gist The DiDi-Merging framework is a slim dynamic merging framework that leverages differentiable rank allocation to balance shared and expert parameters, achieving near-original accuracy with minimal overhead and outperforming prior dynamic baselines across vision, language, and multimodal tasks<ref:2605.18904#pg2>.

How it works

DiDi-Merging addresses the two failure modes in existing dynamic merging methods by balancing shared and task-specific components through differentiable rank allocation<ref:2605.18904#pg3>. It formulates parameter budgeting as differentiable rank optimization in low-rank modules and introduces a data-free refinement step to recover task fidelity<ref:2605.18904#pg5>. The process involves three key steps: (1) Task vectors are statically merged into a shared knowledge vector, with per-task residuals as expert vectors (exact but parameter-heavy)<ref:2605.18904#pg6>, (2) Low-rank decomposition recasts parameter budgeting as a differentiable rank allocation problem, learned end-to-end against a task-vector reconstruction objective<ref:2605.18904#pg7>, and (3) The truncated low-rank parameters are refined in a data-free pass to recover task-specific performance<ref:2605.18904#pg8>.

Key Mechanisms

((a) Heavy Shared Knowledge (b) Heavy Experts Knowledge (c) Balanced Shared–Expert)

The framework targets the balanced regime where both shared and expert components are kept small, which is achieved via differentiable rank optimization<ref:2605.18904#pg3>. This contrasts with prior methods that either keep a full-size shared model with tiny per-task experts or allocate excessive capacity to experts<ref:2605.18904#pg4>.

((b) Heavy Shared Knowledge (a) Heavy Experts Knowledge (b) Balanced Shared–Expert)

The core of the method involves decomposing both shared and expert knowledge into low-rank modules<ref:2605.18904#pg5>. The shared matrix Ms is obtained via static merging of the task vectors Tt, e.g., by averaging<ref:2605.18904#pg6>, and each expert matrix Mt is defined as the residual relative to the shared component<ref:2605.18904#pg6>. This decomposition is then subjected to Singular Value Decomposition (SVD) to obtain low-rank modules M˜(r)s and M˜(r)t<ref:2605.18904#pg5>.

Optimization and Refinement

((b) Differentiable Optimization of Ranks)

To efficiently determine the optimal rank of each shared and expert module in a layer, DiDi-Merging leverages a differentiable approach based on the smooth truncation of the singular values from the SVD of the matrix M<ref:2605.18904#pg5>. A smooth truncation function σ˜i = σi / (0.5 · tanh β(r − i) + 0.5) is used, where r is a learnable continuous rank parameter<ref:2605.18904#pg5>. Jointly minimizing a task reconstruction loss and a compression penalty yields a slim allocation for each module<ref:2605.18904#pg8>.

((c) Low-rank Module Update)

After obtaining the optimal rank allocation, LoRA-style modules are instantiated to recover performance lost from rank truncation<ref:2605.18904#pg8>. The task reconstruction loss Ltask is defined as the mean squared error over all tasks and layers between the reconstructed vector τ˜t,l and the original task vector τ t,l<ref:2605.18904#pg8>. This loss is minimized using a lightweight, data-free optimization with Adam to update only the LoRA factors At,lBt,l<ref:2605.18904#pg8>.

Performance and Efficiency

DiDi-Merging achieves superior performance across vision, language, and multimodal tasks by matching prior dynamic baselines at 1.24× the single-model parameter count and surpassing them at 1.4×<ref:2605.18904#pg8>. Across vision, language, and multi-modal (audio / video / point-cloud) tasks, DiDi-Merging retains ≥ 98% of fine-tuned accuracy at 1.24× the single-model parameter count<ref:2605.18904#pg8>. In LLM merging settings, DiDi-Merging consistently outperforms prior methods by +5.4% on average, with the largest gains on instruction-following benchmarks<ref:2605.18904#pg8>.

Comparative Advantages

The paper makes three contributions: (1) We identify two failure modes in existing dynamic merging methods (heavy shared and heavy expert), both stemming from one-sided shared/expert allocation<ref:2605.18904#pg8>. (2) We propose DiDi-Merging, a dynamic merging framework with differentiable rank allocation: ranks are jointly optimized for shared and per-task expert components against task-vector reconstruction targets, without access to original training data<ref:2605.18904#pg8>. (3) Across vision, LLM, and multimodal benchmarks, DiDi-Merging achieves near-lossless performance (≥ 98% retention) at a 1.24× parameter budget, smaller than every prior dynamic baseline<ref:2605.18904#pg8>.

Computational Cost and Scaling

The framework is computationally efficient; DiDi-Merging achieves the highest accuracy at a modest compute footprint (1.1 GB GPU memory; ∼10 min Find-Optimal-Rank + ∼4 min Low-rank Module Update), well below WUDIMerging’s 4.0 GB<ref:2605.18904#pg9>. The storage cost is decomposed into backbone, router, shared low-rank, and per-task experts<ref:2605.18904#pg8>. The analysis shows that the optimal ranks scale with module size and that the per-module ratio between the two budgets is roughly constant (∼1.7×), consistent with the linear scaling of the budget delta from 0.24× to 0.4×<ref:2605.18904#pg9>.

Broader Impact

DiDi-Merging reduces the storage and serving cost of multi-task systems, lowering the deployment barrier for collections of specialised models<ref:2605.18904#pg8>. This offers a path to deploy expert ensembles without retraining or exposing underlying patient data, which is particularly promising in high-stakes multimodal deployments such as medical diagnosis<ref:2605.18904#pg8>. The framework inherits the biases of its source checkpoints, but its efficiency allows for deployment in low-resource personalization and on-device multi-task inference<ref:2605.18904#pg8>.

Limitations

DiDi-Merging has three known limitations: First, we evaluate under the ideal router setting (task identity at inference; §3.1); learning a router from raw inputs is orthogonal future work, and the reported numbers should be read as an upper bound on capacity utilization<ref:2605.18904#pg8>. Second, while merging is data-free, it requires storing the fine-tuned task vectors Tt (and thus the fine-tuned checkpoints), so the pipeline cannot run in storage-constrained settings where checkpoints are not retained<ref:2605.18904#pg8>. Third, the Find-Optimal-Rank stage is sequential per task (∼10 min on 8 tasks; Tab. 7) and is the dominant wall-clock cost<ref:2605.18904#pg8>.

Data and Code Availability

The code will be released upon acceptance<ref:2605.18904#pg32>. All datasets used in the paper are public benchmarks (image classification, GLUE, AudioCaps, ScienceQA, ModelNet40, etc.); the appendix lists their sources<ref:2605.18904#pg32>. The method, hyperparameters, and baselines are fully described in the main paper and appendix so that the experiments can be reproduced from the manuscript alone<ref:2605.18904#pg32>. All datasets used in the paper are public benchmarks (image classification, GLUE, AudioCaps, ScienceQA, ModelNet40, etc.); the appendix lists their sources<ref:2605.18904#pg32>. The method, hyperparameters, and baselines are fully described in the main paper and appendix so that the experiments can be reproduced from the manuscript alone<ref:2605.18904#pg32>. The code will be released upon acceptance<ref:2605.

1. Precise Parameter Allocation for Efficiency: The system can balance shared and expert parameters using differentiable rank allocation to match prior dynamic baselines at 1.24x or surpass them at 1.4x, substantially more compact than methods requiring two times the storage of a single fine-tuned model.

2. Data-Free Task Fidelity Recovery: The system employs a data-free refinement step to recover task fidelity after low-rank truncation by using the stored task vectors as reconstruction targets, ensuring near-lossless performance at 1.24x the parameter count.

3. Adaptive Knowledge Representation: Instead of heavy shared or heavy expert regimes, the system represents both shared and expert knowledge as low-rank modules with flexible rank allocation that adapts capacity to task similarity via an optimal shared rank ratio learned through differentiable optimization.

4. Robust Performance Across Modalities: The framework achieves near-lossless performance (≥ 98% retention) across vision, language, and multimodal tasks, including MLLMs like audio and video, demonstrating that shared knowledge across modalities provides additional information that enhances model performance.

5. Scalable PEFT Merging: The method extends seamlessly to PEFT settings where each fine-tuned model is parameterized as LoRA adapters; it merges these adapters dynamically while preserving near-lossless performance with a slim parameter footprint.

Model merging consolidates independently fine-tuned checkpoints without joint training or access to task data. Dynamic methods retain task-specific capacity, but must divide a limited storage budget between shared and expert parameters. We propose DiDi-Merging, which jointly allocates low-rank capacity to these components through differentiable rank selection, followed by data-free parameter-space refinement. On the eight-task ViT-B/32 benchmark, the small configuration reaches 89.3% average accuracy (98.7% normalized retention) at 1.24x the storage of one backbone, including that backbone; the medium configuration reaches 89.8% at 1.4x. Removing refinement reduces the medium configuration by 4.3 percentage points, while static-merge plus uniform-recovery controls are 4.7-8.9 points below the small configuration. Additional evaluations cover language and multimodal models at their separately reported budgets. The main results assume known task identity; practical routing and accuracy under larger task counts at fixed capacity remain distinct evaluation questions. Code is available at https://github.com/WanyuGroup/NeurIPS2026-DiDi-Merging.