SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
cs.LG, cs.AI
Submitted: 2026-02-02
Updated: 2026-08-26
Comments: Accepted to ICML 2026. Code is available at https://github.com/LAMDA-CL/Prism
Code: https://github.com/LAMDA-CL/Prism
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal
Terminology
Abstract
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. We also introduce a new benchmark to evaluate MCIT with long task sequence, and extensive experiments demonstrate SAME's SOTA performance. Code is available at https://github.com/LAMDA-CL/Prism.
Sources
- Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models
- CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning
- LLaVA-c: Continual Improved Visual Instruction Tuning
- Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
- IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
- OASIS: Online Sample Selection for Continual Visual Instruction Tuning
- Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
- Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
- LLaMA: Open and Efficient Foundation Language Models
- Large Continual Instruction Assistant
- LoKI: Low-damage Knowledge Implanting of Large Language Models
- HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning
- Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- How to Teach Large Multimodal Models New Skills
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
- LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models
- Instruction-Following Evaluation for Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks