Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

The gist: DRAPE synthesizes continuous instance-specific soft prompts for Multimodal Continual Instruction Tuning by deriving prompt queries from textual instructions and cross-attending to visual patch features, which improves performance over static task-level prompting while mitigating catastrophic forgetting through null-space gradient projection and CLIP-based prototype routing.

Motivation

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks, which is the challenge addressed by Multimodal Continual Instruction Tuning (MCIT) aiming to acquire new capabilities while limiting catastrophic forgetting Existing MCIT methods mainly follow a module-composition paradigm where they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, reasoning demands, and instruction–image interactions. This motivates instance-level adaptation to individual query–image pairs rather than only selecting or combining task-level modules.

DRAPE Framework Components

DRAPE is a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. It contains three key components:

1. A cross-modal prompt generator Gt that synthesizes Lp instance-specific soft prompt vectors Pi = Gt(wi, ui) conditioned on both the visual and textual features of each input. This generation process involves Instruction-Aware Query Initialization, where instruction embeddings are projected into a latent space to compute initial pooled query matrices H¯, which are then used as queries in a multi-head attention layer over the full instruction features. Subsequently, Vision-Guided Prompt Synthesis uses cross-modal attention between the instruction queries Q and projected visual features hv to produce the instance-specific soft prompts Pi = fhead(R), which are prepended to the frozen LLM input sequence z, yielding z' = [Pi; z].

2. Null-Space Gradient Projection for Projector Preservation This mechanism constrains the gradient updates of the shared visual projector π using an approximate null-space projection derived from empirical feature subspaces observed in previous tasks to reduce interference with previously observed feature subspaces. This is achieved by maintaining a cumulative second-moment matrix M(t) and decomposing it via singular value decomposition to identify the dominant feature subspace, and then constructing a projection matrix Π(t) from the complementary basis V⊥, which is applied to the gradients of each weight matrix in π during training.

3. CLIP-Guided Prototype Routing This mechanism selects the most relevant task-specific generator at inference without task labels by using frozen CLIP embeddings as a shared semantic space. A task prototype ct is registered after each task, refined via a classification objective that pulls current-task embeddings closer to ct than to previously stored prototypes, and the router selects the generator Gt∗ by scoring each task st = cos(e˜, ct) and selecting t∗ = arg max t∈T st.

Continual Learning Mechanisms

To mitigate forgetting during sequential updates, DRAPE combines these components with two complementary mechanisms. First, null-space gradient projection is applied to the shared projector so that updates are biased toward directions that minimally interfere with previously observed feature subspaces. Second, CLIP-based prototype routing selects the most relevant task-specific generator without task labels at inference. The training objective is the standard autoregressive language modeling loss optimized over the tunable parameters Θt = Gt, π.

Experimental Results and Analysis

Extensive experiments on MCIT benchmarks show that DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines. On the CoIN benchmark, DRAPE achieved an average accuracy of 67.48, outperforming the strongest baseline ProgLoRA (59.09) by 8.39 points and obtaining the best results on four tasks: TextVQA, ImageNet, VizWiz, and Grounding. The null-space projection analysis shows that when previous-task features are well captured by the retained principal subspace, their components in the complementary subspace are small, and the projected update has a limited first-order effect on their layer outputs. Qualitative case studies on GQA, VQAv2, and OCR-VQA illustrate that instance-specific prompt generation can better accommodate shifts from simple grounding to finer spatial reasoning and compositional attribute judgment within the same task.

Efficiency and Limitations

In terms of efficiency comparison on the CoIN benchmark, DRAPE updates 32M parameters per task, which is substantially smaller than MoELoRA and CODA-Prompt while also requiring less training time in our setup. The storage overhead for each new task is relatively low, adding only one frozen task-specific generator, occupying about 21.04 MB in our setting. A limitation noted is that while DRAPE is developed and validated in the prompt-tuning setting considered in this paper, it has not yet extended the same instance-specific adaptation idea to other lightweight adaptation paradigms such as LoRA.

Conclusion

DRAPE addresses multimodal continual instruction tuning through three complementary mechanisms: a cross-modal prompt generator for input-conditioned prompt synthesis, a projection-based constraint that reduces interference in the shared visual projector, and a CLIP-guided prototype router for practical task-specific generator selection without task labels. Extensive experiments on the CoIN and UCIT benchmarks demonstrate that DRAPE achieves strong performance across diverse multimodal tasks and maintains competitive continual learning behavior. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands. The results show that DRAPE improves forgetting resistance while maintaining strong overall continual performance throughout the task stream. The method is lightweight both during training and in long-term per-task expansion. This framework provides a more flexible adaptation mechanism that better matches sample-specific multimodal demands <ref:2605.

1. Dynamic Cross-Modal Prompt Generation (DRAPE) allows Multimodal Large Language Models (MLLMs) to acquire new capabilities while limiting catastrophic forgetting by synthesizing continuous instance-specific soft prompts from textual instructions and cross-attending to visual patch features. This enables adaptation to individual query image pairs rather than only selecting or combining discrete task-level modules.

2. Null-space gradient projection constrains the gradient updates of the shared visual projector so that updates are biased toward directions that minimally interfere with previously observed feature subspaces, thereby preserving earlier-task mappings in the shared projector. This mechanism is formalized by projecting gradients via a projection matrix derived from a cumulative second-moment matrix to restrict updates to the complementary subspace of dominant feature directions.

3. CLIP-guided prototype routing enables task-label-free generator selection at inference by comparing input routing features against stored prototypes built from fused CLIP text-image features. This allows the model to select the most relevant task specific generator without task labels, leading to improved adaptation to diverse query image pairs in a rehearsal-free manner.