Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

arXiv:2610.22075 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones".

Jane: The gist The proposed approach outperforms unpruned baselines while storing 23.3% fewer language-model parameters by retrofitting depth recurrence into a pretrained vision-language model Retrofitting Mechanism The core of the work…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper titled "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones." Basically, the authors are taking a vision-language model and trying to make it smaller by using depth recurrence.

Jane: That sounds like they're trying to take a big model, right, and strip away some of the unnecessary parts so it runs faster or uses less memory. What is their main idea for how they do that?

Tom: They adapt the idea of depth recurrence from language models—where you loop over certain layers—and apply it to visual understanding within a vision-language model. The goal is reducing the parameter footprint of this pretrained vision-language model, which is a big deal for deployment.

Lu: What they're doing specifically is pruning layers from the language backbone of the VLM and then looping over just a subset of those remaining layers. They call this strategy "prune-then-loop retrofit" <ref:2610.22075#pg3>.

Meng: So, if you prune some layers, you're essentially cutting out parts of the original model architecture, and then you try to make those remaining parts work in a loop instead of just passing through them once. How big is this parameter reduction they are aiming for?

Tom: They claim that by using this method, they achieve twenty-three point three percent fewer language parameters compared to the unpruned baseline when trained with the same recovery data. That's a solid number <ref:2610.22075#pg1>.

Jane: Twenty-three point three percent is significant, especially when you're talking about a vision-language model that already has a large language backbone. But how do they fix the performance drop that comes with pruning? That seems like the hardest part for any retrofitting process.

Lu: To fix it, they propose a three-stage recovery training curriculum <ref:2610.22075#pg1>. First, they use a language-only pretraining stage to recover the performance of the language backbone while freezing everything else.

Tom: That makes sense; you need to repair the language part before you mess with the visual alignment. After that, they move to aligning that vision-language projector and then do a full fine-tuning using visual instruction tuning.

Meng: So it's not just one training step; it’s a whole sequence of recovery stages designed to patch up the damage from pruning <ref:2610.22075#pg3>. What about how they handle the knowledge transfer between these stages?

Jane: The paper mentions that in the first and last stages, they distill the outputs of the pretrained non-recurrent model into their recurrent model, which is supposed to further improve recovery. That sounds like using what you already have to help fix what you've broken <ref:2610.22075#pg1>.

Lu: They also incorporated a knowledge distillation term into stages one and two to limit the drift of the language backbone from its original representation, using the pretrained non-recurrent model as a teacher. That helps keep things stable during training <ref:2610.22075#pg3>.

Tom: Now for some design choices, they did an ablation study on things like the adapter between recurrences. They found that a linear adapter yields the best performance among tested variants, outperforming gated and Parcae adapters <ref:2610.22075#pg3>.

Paper summary: Jane: So it's not just about *if* you loop, but *how* you loop—the specific mechanism connecting the layers in the loop matters a lot for how well it recovers performance. And they also found that a larger recurrent block performs better at equal compute <ref:2610.22075#pg3>.

Meng: From a deployment standpoint, that's important because it tells us we can tune the architecture to get better results without just throwing more compute at it. But what about inference? How does this change when the model is actually running in production?

Tom: They looked at KV cache sharing during inference and found that keeping a separate cache for each recurrence block is necessary, because sharing costs accuracy <ref:2610.22075#pg3>. They also introduced an adaptive halting criterion based on KL divergence to skip computation when it's not needed.

Jane: So, while the training recovery is complex and involves multiple steps, they are also thinking about how this model will behave once it’s actually in use for generating text or answering questions. That adds a layer of practical engineering <ref:2610.22075#pg3>.

Lu: The paper also found that a model trained at a fixed depth only works at the depth it was trained at, but one trained with sampled recurrence works at other depths, though its performance doesn't improve beyond the trained depth <ref:2610.22075#pg3>.

Tom: That means you can train it in a flexible way, but you still have to respect the original training depth for peak performance. But they also found that sharing KV cache across recurrences during inference costs accuracy, which is another practical constraint <ref:2610.22075#pg3>.

Jane: So the main point here is that this retrofitting approach outperforms a recurrent language model extended to a VLM on the same data, and it’s not just about adding a vision encoder to an existing recurrent model <ref:2610.22075#pg1>.

Tom: What's really interesting is where they see the gains. They show that this proposed model gains on reasoning-oriented benchmarks, specifically MMBench and RealWorldQA <ref:2610.22075#pg3>.

Meng: So it’s not a general boost across the board for every visual task, but it’s particularly helpful when the task requires complex reasoning capabilities. That helps us understand where this kind of efficiency gain is most valuable in real-world applications <ref:2610.22075#pg3>.

Jane: And on the other hand, they found that its performance on basic visual understanding benchmarks remains on par with the unpruned baseline, which they attribute to those datasets usually having single-step question types <ref:2610.22075#pg3>.

Tom: So in summary, this paper introduces the first prune-then-loop retrofit of a pretrained VLM that outperforms unpruned baselines while storing twenty-three point three percent fewer language parameters <ref:2610.22075#pg1>.

Lu: The conclusion is that retrofitting recurrence directly into a pretrained VLM results in better performance than introducing a vision encoder into a recurrent language model to create a recurrent VLM <ref:2610.22075#pg3>.

Jane: It seems like the core idea isn't just about size reduction, but finding a way to reuse the existing structure of the pretrained model in a more efficient way for visual tasks. That’s what this paper is about <ref:2610.22075#pg1>.

Conclusion: Tom: So we've seen how they took depth recurrence from language models and retrofitted it into a vision-language model to make it smaller, right?

Jane: It sounds like they’re trying to reuse an existing structure instead of building a new one from scratch. What are the authors of this paper, Tom?

Tom: The paper is "Retrofitting Recurrent Vision-Language Models from Pretrained Backbones," and I see Lu and Jane on today.

Lu: I'm Lu, and we looked at how they prune layers to create a recurrent block, which is kind of interesting because it’s a structural change.

Jane: So the main point here is that this method outperforms just adding a vision encoder to an old recurrent language model <ref:2610.22075#pg1>. What does that mean for someone who just listens to the show?

Tom: It means they found a way to get better results with less language parameter count, specifically twenty-three point three percent fewer parameters on TinyLLaVA-one point 4B, compared to the unpruned baseline <ref:2610.22075#pg1>.

Meng: Twenty-three point three percent is a solid number for efficiency, but I wonder if that performance gain is real or just on a specific test set, right?

Lu: The authors did ablation studies on the design choices, and they showed that the linear adapter between recurrences is actually the best one tested <ref:2610.22075#pg3>.

Jane: So it’s not just about looping layers; it’s about *how* you connect them in that loop that makes a difference, and they even found a larger recurrent block works better than a smaller one for the same amount of compute <ref:2610.22075#pg3>.

Tom: And they showed gains on reasoning-oriented benchmarks like MMBench and RealWorldQA, but basic visual understanding performance stayed the same, which makes sense given how those tasks are usually structured <ref:2610.22075#pg3>.

Jane: So what does this suggest about building future AI models?

Lu: It suggests that retrofitting recurrence directly into a pretrained vision-language model is better than just tacking on a vision encoder to an existing recurrent language model <ref:2610.22075#pg3>.

Tom: Exactly. If you want efficiency without completely starting over, reusing the backbone structure in this recurrent way seems like the path forward for some of us.

Madhav Iyengar, Andreas Geiger, Hilde Kuhne

University of Tubingen AI Center

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: Project page: https://madhavesdios.github.io/recurrent-vlms

Project page: https://madhavesdios.github.io/recurrent-vlms

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist The proposed approach outperforms unpruned baselines while storing 23.3% fewer language-model parameters by retrofitting depth recurrence into a pretrained vision-language model Retrofitting

Key concepts

Depth Recurrence Retrofitting: This technique involves taking a pretrained vision-language model (VLM), removing some of its layers (pruning), and then looping over the remaining subset. A linear adapter is used at each step to merge the output of the current block with the representations from previous layers, effectively reintroducing recurrence into a pruned structure.
Prune-then-Loop Retrofit: This describes the structural modification where layers are partitioned into four groups: a prelude (P), a recurrent block (B), and a coda (C). The model then applies the recurrent block 'r' times, using an adapter to connect it back to the prelude. This process modifies the original model's architecture by selectively keeping only necessary layers for recurrence.
Three-Stage Recovery Training: To fix performance loss from pruning, a three-step training curriculum is used. First, text-only pretraining repairs the language backbone. Second, the vision-language projector is retrained to ensure image and text align correctly. Finally, full fine-tuning with visual instruction tuning restores overall model capability.
Knowledge Distillation (KD): Knowledge distillation is used during recovery training stages to prevent the language backbone from drifting too far from its original pretrained state. The output of the large, non-recurrent teacher model is used to guide and constrain the training of the smaller, recurrent model.

Terminology

Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed approach outperforms unpruned baselines while storing 23.3% fewer language-model parameters by retrofitting depth recurrence into a pretrained vision-language model

Retrofitting Mechanism

The core of the work involves adapting depth recurrence to visual understanding by pruning layers from the language backbone of a Vision Language Model (VLM) and looping over a subset of the remaining layers. This strategy is formally described as prune-then-loop retrofit, where intermediate blocks are removed, and a contiguous subset of surviving layers is designated as a recurrent block. The process modifies the original model's structure by partitioning the m layers into four consecutive groups of sizes p + d + b + c = m, deleting the d layers to form a prelude block P, a recurrent block B, and a coda block C. The retrofitted model applies the recurrent block r times using an adapter A that merges the current output with the prelude's output. This operation is expressed as F r p/b/c:= C ◦ (B ◦ A) r-1, P >.

Three-Stage Recovery Training

To recover performance lost due to pruning and misalignment, a three-stage training curriculum is proposed. First, the model undergoes a text-only training stage S0 to repair the language backbone by leveraging language pretraining while freezing all other components. Second, in stage S1, the vision-language projector is retrained to restore image-text alignment. Finally, stage S2 involves full fine-tuning of the model via visual instruction tuning using visual data. A knowledge distillation (KD) term is incorporated into stages S0 and S2 to limit the drift of the language backbone from its original representation, using the pretrained non-recurrent model as a teacher.

Design Choices and Ablations

Extensive ablation studies were conducted to quantify the impact of various design choices on performance. Key findings include:

** Adapter Design The linear adapter Alin yields the best performance among tested variants, outperforming gated and Parcae adapters. The no adapter variant A∅ performs nearly as well as Alin with a performance gap of only 0.73 pp.**

** Layer Allocation A larger recurrent block performs better than other variants given the same compute setting, specifically the 4/8/4 allocation outperforms a 6/4/6 allocation at r=11 by 3.19 pp.**

** Training Stages The text-only stage S0 is crucial for recovery training, as removing it reduces performance by 3.68 pp on average, primarily affecting reasoning-oriented datasets.**

** Knowledge Distillation Knowledge distillation in stage S2 always improves the overall performance, and having both image-text as well as text-only data leads to the most significant improvement, improving over the undistilled version by 1.48 pp.**

Inference Considerations

The paper also evaluates inference efficiency through KV cache sharing across recurrences. The authors tested two methods for sharing the cache across passes: keeping n copies of the block’s cache per token (K-sharing) or letting every pass reuse keys and values from the first pass (KV-sharing). They found that keeping a separate cache per recurrence in their best setting is necessary, as sharing costs accuracy at inference without retraining. Furthermore, an adaptive halting criterion based on KL divergence is introduced to enable compute-skipped generation for improved efficiency.

Performance Summary

The proposed Retrofitted Recurrent VLM (RR-VLM) outperforms the unpruned baseline while having 23.3% fewer language-model parameters, with gains concentrating on reasoning-oriented benchmarks such as MMBench and RealWorldQA. The performance on basic visual understanding benchmarks remains on par with the baseline, which is attributed to the fact that those datasets usually have single-step QA type questions. The retrofitted model also outperforms a recurrent language model extended to a VLM by achieving higher average accuracy.

Conclusion

The work introduces the first prune-then-loop retrofit of a pretrained VLM evaluated on VQA, demonstrating that retrofitting recurrence directly into a pretrained VLM outperforms adding a vision encoder to a pretrained recurrent language model. The proposed method yields the best performance among all evaluated training strategies, and its gains are primarily due to the recurrence itself rather than the recovery training. The final conclusion is that retrofitting recurrence into a pretrained VLM results in better performance than introducing a vision encoder into a recurrent language model to create a recurrent VLM.

The gist The proposed approach outperforms unpruned baselines while storing 23.

Improvements for AI systems

Bold header: Prune-then-loop retrofit of pretrained VLMs

This system can achieve a 23.3% reduction in language parameters while outperforming unpruned baselines on reasoning-oriented benchmarks, specifically showing a clear trend of performance increase on more reasoning-oriented datasets such as MMBench, RealWorldQA, SEED-Bench, AI2D and CV2D.

Bold header: Three-stage recovery training curriculum

The proposed three-stage curriculum ("first we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning) ensures that the text-only and the instruction-tuning stage S0 is necessary for the recovery," leading to performance gains over baselines.

Bold header: Optimized linear adapter design

The study found that the linear adapter Alin (Eq. 3) achieves the highest performance compared to variants like Asc and Apc, indicating that using a specific linear layer for merging the prelude and recurrent block output is crucial for maximizing performance gains in this retrofit strategy.

Bold header: KV-cache sharing at inference

The model can be deployed with n=6 (ours) cache sharing, which results in a significantly larger cache (2.55× the KV cache of the base model) compared to n=1, and is demonstrated to outperform it on various benchmarks, although this comes at an explicit cost of 0.73 pp for sharing keys and values during training.

Bold header: Adaptive halting criterion for inference

The system supports compute-skipped form generation using the adaptive halting criterion (Eq. 5), allowing the model to terminate the forward pass dynamically at step k⋆ = min[k ≥ 2: KLsoftmax(yk) softmax(yk−1) < ε], which enables adaptive halting criterion (Schuster et al., 2022; Geiping et al., 2025) to balance accuracy and compute.

Abstract

Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understanding, using depth recurrence to reduce the parameter footprint of a pretrained vision-language model (VLM). To this end, we prune layers from the language backbone of the VLM and loop over a subset of the remaining layers. At each step of the loop, a linear adapter merges the block's output with the token representations from the preceding layers and projects the fused representation back into the block. As the pruning impacts the original model performance, we propose a three-stage recovery training: first, we leverage a language-only pretraining to recover the performance of the language backbone, followed by an alignment of the vision-language projector, and a full fine-tuning of the model via visual instruction tuning. In the first and the last stage, we distill the outputs of the pretrained non-recurrent model into the recurrent model, which further improves the recovery. We evaluate the proposed approach on the TinyLLaVA-1.4B backbone without introducing any new vision-language data during retrofitting. Our experiments demonstrate that the retrofitted recurrent model outperforms the unpruned baseline trained with the same recovery data while storing 23.3% fewer language parameters. We further show that the proposed retrofitting recurrence outperforms the setup of converting a recurrent LM into a VLM on the same data. Additionally, our ablations show how design choices, such as the adapter between recurrences and a larger recurrent block, lead to the improved performance of our approach. Finally, we find that our proposed model gains on reasoning-oriented benchmarks, whereas its performance on basic visual understanding benchmarks remains on par with the baseline.