LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models

arXiv:2605.10641 · cs.CV, cs.AI · Submitted 2026-05-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models".

Jane: Here's the gist of the paper. The gist: The proposed bottom-up cascaded knowledge distillation (CKD) framework introduces one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over; this is inspired by human formal education systems.

Tom: Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question Answering (VQA), but their memory and compute requirements remain a concern for practical deployment Knowledge Distillation (KD) is a promising class of techniques for mitigating this concern, where knowledge from a high-capacity Teacher network is transferred to a considerably smaller Student network However, the capacity gap between the two networks is both a blessing and a curse: the smaller the Student network, the better its efficiency, and the larger the Teacher, the more knowledge it carries; yet, beyond a point, the larger capacity gap between the two leads to worse knowledge transfer Proposed Method: Bottom-up Cascaded Distillation Scheme The proposed method involves a bottom-up cascaded distillation scheme where three VLMs fs, fa and ft have the role of the Student, the TA and Teacher, respectively The scheme is performed in two stages to gradually train the Student 1) Stage 1: TA →Student distillation.

Jane: In this stage the TA and the dataset D are used to train the Student via distillation The specific KD approach applied at this stage should utilize an appropriate loss, to account for the knowledge transfer of both textual and visual information of the TA to the Student 2) Stage 2: Teacher →Student distillation.

Tom: Stage 1 is repeated, replacing the TA with the stronger Teacher model, and the original Student with the distilled Student model coming out of Stage 1 For simplicity of illustration, we consider only one TA, but the proposed approach can be extended to more than one by introducing additional stages Training Components and Distillation Details The training utilizes the LLaVA

17: architecture for the Student/TA/Teacher VLMs, where each VLM consists of a pretrained vision encoder and LLM backbone, and a vision-language connector The TA and Teacher models are (pre-)trained using the above dataset for general-purpose visual and language understanding The LLaVA-KD

7: approach is used to train the Student at each stage of CKD, which consists of three steps: Distillation pretraining (DPT), Supervised finetuning (FT), and Distillation finetuning (DFT) In the DPT step, the loss used is Ldf t = Lrg + τ1Ltd + τ2Lvd + τ3Lvc The losses are defined as: Lrg = −k X−1 i=1 Xc j=1 yi+1,j ln pi,j, Lvd = Xm i=1 Xc j=1 si,j ln si,j zi,j, Ltd = X k i=m+1 Xc j=1 si,j ln si,j zi,j, and Lvc = 1 − Pm i=1 a T i bi qPm i=1 a T i ai qPm i=1 b T i bi Theoretical Analysis The theoretical analysis is based on statistical learning theory, expressing the classification error of the Student fs as R(fs) − R(f) ≤ O FsC nas + ϵs The cascaded distillation process provides an alternative expression for the rate at which the Student learns the truth function as R(fs¯) − R(f) = R(fs¯) − R(ft) + R(ft) − R(f) ≤ O Fs¯C nast̄ + ϵst + O FtC nat + ϵt This derivation is based on combining Eqs.

Paper summary: Tom: So, LLaVA-CKD Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models is basically this layered approach where you use a sequence of Teacher and Assistant models to gradually improve the Student model's performance.

Jane: It’s about using that structured teaching process, inspired by education, to manage the capacity gap problem in knowledge distillation.

Lu: The paper shows that by introducing these intermediate Teacher models and following a two-stage strategy, you can improve both the model's performance and create a better dataset for training it in one stage.

Meng: It means we can potentially deploy smaller vision language models that perform much better than they would otherwise be, which is good for scaling things down.

Jane: The authors found this works well across seven different VQA benchmarks, showing gains of zero point nine percent and one percent on the LLaVA-KD baselines in certain scales.

Tom: They used a very specific framework involving three steps: Distillation pretraining DPT, Supervised finetuning FT, and Distillation finetuning DFT, which is what makes this method concrete.

Lu: And they did a theoretical analysis using statistical learning theory to prove that the performance gain happens in the asymptotic regime.

Jane: Overall, it gives us a concrete framework for mitigating the issue where a large capacity gap between Teacher and Student can hurt knowledge transfer.

Tom: It's about making distillation more structured by introducing intermediate models instead of just one large teacher model.

Conclusion: Tom: So, we're wrapping up this deep dive on LLaVA-CKD Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models. It’s about how you can get a smaller AI model to learn as well as a much bigger one by using a chain of smaller teachers along the way.

Jane: Right, so if I'm listening, the main thing is that they're not just doing one big transfer from teacher to student. They’re introducing these intermediate models, like TA and Teacher, that progressively prepare the Student for the final step.

Lu: Exactly. The core idea is inspired by how formal education works—you move through levels of learning where you build up your knowledge before moving to a more advanced teacher. It’s about making that capacity gap manageable by bridging it piece by piece.

Meng: From my side, I’m interested in the practical implication here. If we can do this cascading, it might mean we can deploy these vision language models on much less compute while keeping the performance high enough for real-world tasks.

Tom: It's definitely a structured way to look at distillation that moves beyond just one big teacher model. The authors showed that this progressive reduction of the gap actually improves the knowledge transfer rate, which is something we need to see more of in deployment scenarios.

Jane: And they backed that up with experiments on seven different benchmarks, showing gains on LLaVA-KD models when comparing them to standard Teacher-Student setups. It’s concrete evidence that this cascading approach isn't just theoretical math; it actually works in practice for these VQA tasks.

Lu: The theoretical analysis based on statistical learning theory supports that idea too, showing how the error rate of the Student learns the truth function through this chain of teachers in a specific mathematical way. It justifies why you need that intermediate step.

Meng: I wonder what happens when you try to scale this up? If you have, say, three different Teacher models at different stages, does that just make the pipeline more complex to manage at scale?

Tom: That’s a good question for the next round. But looking at the paper's conclusion, they are focused on extending this idea to more complex architectures and combining different VLM families in this cascade. They're aiming for a more robust way to close that gap between a very strong teacher and a very small student model.

Jane: So it’s not just about making one distillation step better, but building an entire system of learning stages. It shifts the focus from one big transfer to a carefully orchestrated sequence of smaller transfers. **

Sound of light, upbeat radio outro music begins to fade in: **

Nikolaos Gkalelis, Vasileios Mezaris

CERTH-ITI

cs.CV, cs.AI

Submitted: 2026-05-11

Updated: 2026-10-20

Comments: IEEE Access, vol.14, 2026

DOI: 10.1109/ACCESS.2026.3744200

Code: https://github.com/IDT-ITI/LLaVA-CKD

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: The proposed bottom-up cascaded knowledge distillation (CKD) framework introduces one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student

Key concepts

Knowledge Distillation (KD): A technique where knowledge from a larger, more capable 'Teacher' model is transferred to a smaller 'Student' model. This helps the Student learn complex patterns and reasoning abilities without needing its own massive size or compute power, making it more efficient for practical use.
Cascaded Knowledge Distillation (CKD): A proposed method where distillation happens in stages using multiple teachers of increasing capacity. The process starts with a smaller teacher transferring knowledge to the student, and then a larger teacher takes over. This progressive reduction of the gap between models is shown to improve knowledge transfer.
Capacity Gap: The difference in model size or complexity between two networks, like a large Teacher and a small Student. The paper argues that while a large gap can hurt distillation, introducing intermediate teachers helps bridge this gap progressively, allowing the student to learn effectively at each step.

Terminology

Distillation — 4× in this paper · explained in 2 episodes
Knowledge Distillation — 3× in this paper · explained in 15 episodes
Vision-language models — 2× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
Visual Question Answering — 1× in this paper

Summary

The gist: The proposed bottom-up cascaded knowledge distillation (CKD) framework introduces one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over; this is inspired by human formal education systems.

Introduction and Motivation

Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question Answering (VQA), but their memory and compute requirements remain a concern for practical deployment Knowledge Distillation (KD) is a promising class of techniques for mitigating this concern, where knowledge from a high-capacity Teacher network is transferred to a considerably smaller Student network However, the capacity gap between the two networks is both a blessing and a curse: the smaller the Student network, the better its efficiency, and the larger the Teacher, the more knowledge it carries; yet, beyond a point, the larger capacity gap between the two leads to worse knowledge transfer

Proposed Method: Bottom-up Cascaded Distillation Scheme

The proposed method involves a bottom-up cascaded distillation scheme where three VLMs fs, fa and ft have the role of the Student, the TA and Teacher, respectively The scheme is performed in two stages to gradually train the Student

Stage 1: TA →Student distillation. In this stage the TA and the dataset D are used to train the Student via distillation The specific KD approach applied at this stage should utilize an appropriate loss, to account for the knowledge transfer of both textual and visual information of the TA to the Student
Stage 2: Teacher →Student distillation. Stage 1 is repeated, replacing the TA with the stronger Teacher model, and the original Student with the distilled Student model coming out of Stage 1 For simplicity of illustration, we consider only one TA, but the proposed approach can be extended to more than one by introducing additional stages

Training Components and Distillation Details

The training utilizes the LLaVA [17] architecture for the Student/TA/Teacher VLMs, where each VLM consists of a pretrained vision encoder and LLM backbone, and a vision-language connector The TA and Teacher models are (pre-)trained using the above dataset for general-purpose visual and language understanding

The LLaVA-KD [7] approach is used to train the Student at each stage of CKD, which consists of three steps: Distillation pretraining (DPT), Supervised finetuning (FT), and Distillation finetuning (DFT)

In the DPT step, the loss used is Ldf t = Lrg + τ1Ltd + τ2Lvd + τ3Lvc The losses are defined as: Lrg = −k X−1 i=1 Xc j=1 yi+1,j ln pi,j, Lvd = Xm i=1 Xc j=1 si,j ln si,j zi,j, Ltd = X k i=m+1 Xc j=1 si,j ln si,j zi,j, and Lvc = 1 − Pm i=1 a T i bi qPm i=1 a T i ai qPm i=1 b T i bi

Theoretical Analysis

The theoretical analysis is based on statistical learning theory, expressing the classification error of the Student fs as R(fs) − R(f) ≤ O FsC nas + ϵs The cascaded distillation process provides an alternative expression for the rate at which the Student learns the truth function as R(fs¯) − R(f) = R(fs¯) − R(ft) + R(ft) − R(f) ≤ O Fs¯C nast̄ + ϵst + O FtC nat + ϵt This derivation is based on combining Eqs. (10), (16), and noting that ast¯ ≤ at, establishing the validity of Eq. (18) in the asymptotic regime

Experimental Results

The experiments were conducted using seven publicly available benchmarks for VLM evaluation: i) VGA v2 (VGAv2) [43] for scene understanding and compositionality, ii) GQA [44] for general visual understanding and relational reasoning, iii) TextVQA (VQAT) [45] for fine-grained visual recognition and understanding of text within images, iv) ScienceQA-IMG (SQAI) [46] to measure scientific knowledge, v) MMP Perception (MMEP) [47] for comprehensive evaluation of multimodal understanding and reasoning capabilities, vi) POPE [48] for hallucination, and vii) MMMU [49] for multi-disciplinary tasks of college-level education The results show that LLaVA-CKD outperforms the baselines in both scales, with a significant average performance gain of 0.9% and 1% observed against LLaVA-KD with Teacher-Student 3B →0.5B and 3B →1.5B, in the 0.5B and 1.5B scales, respectively

Conclusion

The proposed CKD framework introduces a cascade of KD stages using a TA/Teacher of varying quality to gradually improve the performance of the Student The experimental evaluation on seven publicly available benchmarks and a theoretical study justified the efficacy of the proposed framework Future directions include the application of CKD to Student-Teacher models that are based on more complex VLM architectures (e.g. mixture-of-experts), and combining VLMs of different families in the same cascade chain in order to close the capacity gap between a high-performing Teacher and the Student more effectively

Acknowledgments

This work was supported by the EU’s Horizon Europe programme under grant agreement 101214398

References

[1] C. X. Liang, P. Tian, C. H. Yin, Y. Yua, W. An-Hou, L. Ming, T. Wang, Z.

Improvements for AI systems

Improved Knowledge Distillation Framework: Implement a bottom-up cascaded knowledge distillation (CKD) framework that uses a Teacher and a Teacher Assistant (TA) of intermediate capacity to gradually bring the Student network to higher performance levels, as proposed in Section 3.1. This allows the system to leverage more than one teacher for direct knowledge transfer, gradually closing the gap between the latter two.
Enhanced Model Efficiency: The resulting small-scale Student VLMs can achieve SotA performance on seven standard VQA benchmarks while reducing memory and compute requirements, as shown in Table 1. This allows deployment of powerful vision-language understanding capabilities in resource-constrained environments.
Scalable Performance Gains: The system demonstrates a significant average performance gain of 0.9% and 1% against baseline KD methods on the 0.5B and 1.5B scales, indicating that the proposed CKD approach effectively closes the capacity gap between Student and Teacher networks during distillation.
Competitive Advantage Over Existing Methods: LLaVA-CKD achieves top or second-best performance in six out of seven benchmarks for the 0.5B scale and best performance in five benchmarks for the 1.5B scale when compared against State-of-the-Art methods like TinyLLaVA and other SotA approaches.
Architectural Flexibility: The proposed framework is applicable to other families of networks and can be instantiated with different base KD approaches, such as LLaVA-KD, allowing for the customization of the distillation process for various VLM architectures.