CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
cs.LG, cs.AI
Submitted: 2026-07-20
Updated: 2026-09-25
License: http://creativecommons.org/licenses/by/4.0/
The gist: Resource-constrained deployment requires sustaining large language model (LLM) capabilities as models scale under fixed memory and computation budgets.
Terminology
Abstract
Resource-constrained deployment requires sustaining large language model (LLM) capabilities as models scale under fixed memory and computation budgets. Structured pruning advances this deployment frontier with smaller dense checkpoints, yet deciding what to prune remains bottlenecked by interactions among joint removals. We introduce Cross-Module Co-Pruning Curvature (CoCurve), which formulates structured pruning as set-dependent predictive risk over a unified inventory of attention heads and feed-forward groups. A co-pruning graph built from single-unit forward ablations assigns individual risk to nodes and reinforcement or cancellation to interaction edges, conditioning each decision on the units already removed. We evaluate 6 LLMs (3B--70B) and 3 vision--language models (VLMs) across 3 perplexity corpora, 12 language tasks, and 7 multimodal benchmarks. Across the five-model 20--40% grid and the 70B 10--50% sweep, CoCurve ranks first in 53/60 corpus comparisons (15/15 at 70B); matched-quality interpolation permits 2.2--6.6 more pruning points in 9/10 cases, while CoCurve leads all 6 VLM Avg 7 blocks. After the same lightweight recovery, its retained structures remain strongest through 50% pruning, where the 8B checkpoint regains 10.8 Avg 12 points; physical slicing delivers 1.58 times dense prefill throughput with 41% lower peak memory. Mechanism analysis across 10 LLMs and 7 VLMs finds organized within- and cross-module edge structure; matched low-saliency, high-coupling removals degrade 19/20 capability groups by up to 23.7 points.
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Optimal Brain Connection: Towards Efficient Structural Pruning
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
- The Llama 3 Herd of Models
- LLM Compression by Block Removal with Constrained Binary Optimization
- Mistral 7B
- Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
- T'yr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization
- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
- Efficient LLMs with AMP: Attention Heads and MLP Pruning
- 2SSP: A Two-Stage Framework for Structured Pruning of LLMs
- SlimLLM: Accurate Structured Pruning for Large Language Models
- DarwinLM: Evolutionary Structured Pruning of Large Language Models
- Faster gaze prediction with dense networks and Fisher pruning
- CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information
- Qwen2.5 Technical Report
- LaCo: Large Language Model Pruning via Layer Collapse
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks