ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Code: https://github.com/jianghoucheng/ReCal
License: http://creativecommons.org/licenses/by/4.0/
The gist: Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery.
Terminology
Abstract
Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
Sources
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
- NIRVANA: Structured Pruning Reimagined for Large Language Model Compression
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning
- SlimLLM: Accurate Structured Pruning for Large Language Models
- Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization
- Distilling the Knowledge in a Neural Network
- Entropy-Aware On-Policy Distillation of Language Models
- DistiLLM: Towards Streamlined Distillation for Large Language Models
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
- Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
- ZipLM: Inference-Aware Structured Pruning of Language Models
- Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- LLM Pruning and Distillation in Practice: The Minitron Approach
- A Simple and Effective Pruning Approach for Large Language Models
- DarwinLM: Evolutionary Structured Pruning of Large Language Models
- The LLM Surgeon
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering