Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score

arXiv:2603.23985 · cs.LG · Submitted 2026-03-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Diet Your LLM".

Jane: Here's the gist of the paper. The gist: DIET is a training-free dimension-wise global pruning method that merges task-wise importance scores to produce a single global mask applicable across all tasks, achieving significant accuracy improvements over state-of-the-art structured pruning methods.

Tom: How it works DIET derives a global pruning mask through a twophase process involving task importance scoring and aggregation First, DIET identifies the dimensions that contribute minimally to each task.

Jane: Specifically, for each task t, it profiles the pretrained LLM fθ on N samples from task t to measure activation magnitudes at the output of the MLP layer in each transformer block.

Tom: It then computes a per-dimension importance score a(t)k by averaging absolute activations across all blocks, tokens, and samples.

Paper summary: Tom: So we're looking at this paper now called "Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score". Jane, what's the main idea here?

Jane: Well, Tom, basically the big problem they're tackling is that large language models are getting so massive that deploying them practically is really hard. Structured pruning offers a way to cut down the model by removing whole dimensions or layers but existing methods have these trade-offs.

Lu: Exactly, and those trade-offs are usually about task specificity versus training cost. Task agnostic methods don't adapt to what the model needs for a specific job, but task aware methods need a lot of training time to learn that adaptability <ref:2603.23985#pg1>.

Meng: I hear that often, when you try to make pruning task aware, it becomes really expensive. If you have to train something new for every single use case the model might see, that just isn't scalable for production deployment <ref:2603.23985#pg1>.

Tom: So what DIET proposes is a training-free structured pruning method that combines dimension level detail with task awareness. Jane, can you tell us how they actually achieve this?

Jane: They propose something called DIET which profiles activation magnitudes across tasks using only one hundred samples per task and then uses majority voting to build a single global mask <ref:2603.23985#pg1>. It claims this avoids the need for large costs from pre-computation or training.

Lu: The core idea is that they first identify the dimensions that contribute minimally to each individual task. Then they aggregate these task specific assessments to see which dimensions most tasks agree are uninformative <ref:2603.23985#pg3>.

Meng: But a single mask sounds risky, right? If it cuts something needed for one task, even if it's useless for others, that’s a deployment nightmare. How do they handle that conflict?

Jane: That's where the merging happens. They use majority voting to create this unified decision so dimensions that multiple tasks independently flag as unimportant are likely safe to remove globally <ref:2603.23985#pg3>.

Tom: So it sounds like they aren't trying to be perfect for every single task, but finding a compromise that works well across the board without any training overhead. Jane, what kind of results are they showing?

Jane: They tested this on seven zero-shot benchmarks using two Gemma models called Gemma-two 2B and Gemma-two 9B <ref:2603.23985#pg1>. The numbers they show are pretty telling for their effectiveness.

Paper summary: Lu: They showed that at twenty percent sparsity on the Gemma-two 2B model, DIET improves average accuracy by two point one percentage points over SliceGPT and nine point two percentage points over PuDDing <ref:2603.23985#pg1>. That's a solid comparison against other structured pruning methods.

Meng: When you talk about being competitive, what does that mean for someone who has to deploy this on their own server? Is it just a small gain in accuracy or does it really translate to something useful?

Jane: They also said that on the Gemma-two 9B model, DIET remains competitive with SliceGPT but consistently outperforms PuDDing at both ten percent and twenty percent sparsity levels <ref:2603.23985#pg1>. It seems like it handles different sizes well.

Tom: So we're seeing gains here, especially since the authors are claiming this is training free and doesn't require those huge pre-computation costs we usually see in task aware methods. Lu, what's the deeper implication of that single global mask they construct?

Lu: The real contribution is how they identify dimensions with the highest cross-task agreement. They found that dimensions receiving seven or six votes account for about ten percent of total dimensions, which lets them achieve their target sparsity with strong consensus <ref:2603.23985#pg3>.

Meng: So it’s not just removing random low activations, but finding the ones that are consistently unimportant across many different things the model has to do. That makes sense for deployment because you're betting on broad consensus rather than one task's opinion <ref:2603.23985#pg1>.

Jane: And they showed it generalizes quite a bit too when they tested it on other models like Qwen2 point 5-7B and Phi-four-Mini-Reasoning, maintaining most of the original accuracy under a ten percent sparsity budget <ref:2603.23985#pg1>.

Tom: That cross architecture generalization is interesting because it means you don't need to rewrite the whole pruning process when you switch models. So, what's the final word on where this method stands compared to what we already have?

Lu: It outperforms a recent structured pruning method and a router-based pruning approach across both model sizes and all sparsity levels <ref:2603.23985#pg2>. They also showed it achieves more than twenty percent accuracy increase on zero-shot commonsense reasoning tasks compared to task wise depth pruning algorithms <ref:2603.23985#pg1>.

Paper summary: Meng: If I'm looking at this from a production standpoint, the claim that it reduces FLOPs and peak memory under hard pruning means we might actually see some latency improvements without sacrificing the performance needed for real tasks <ref:2603.23985#pg2>.

Jane: So it seems like this paper is suggesting that you can get task aware compression with modest profiling costs using a simple, activation driven approach rather than needing complex training pipelines <ref:2603.23985#pg1>.

Tom: That's what I found interesting about DIET, the idea of using just one hundred samples per task to guide this whole process. It keeps the initial profiling cost manageable.

Lu: The limitation they mentioned is that their evaluation focused only on seven English centric zero-shot benchmarks and they haven't explored multilingual domains or safety critical tasks yet <ref:2603.23985#pg4>.

Meng: That makes sense, but if it works well on the models Gemma-two 2B and 9B, we have a good starting point for deployment right now <ref:2603.23985#pg1,Gemma-2 2B and 9B>. What I'd watch is how this scales to really complex or safety sensitive applications where those biases might pop up <ref:2603.23985#pg4>.

Jane: So, the big picture here is that we have a new tool for making LLMs smaller and faster for real use cases without needing massive retraining cycles <ref:2603.23985#pg1>. It’s about making these large models accessible to more people in terms of hardware requirements.

Tom: So, the final thought on Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score is that it provides a robust way to compress models by combining task importance scores into one global mask <ref:2603.23985#pg1>.

Jane: It sounds like a solid method for deployment right now because it offers clear accuracy gains over existing techniques with very little extra training work required <ref:2603.23985#pg1>.

Lu: And the way they use majority voting to find that agreement across tasks is a clever way to avoid being too specific to just one narrow task requirement <ref:2603.23985#pg3>.

Meng: I'm just watching how they handle those higher sparsity ratios, because if we push it too far past twenty percent, I expect the accuracy degradation to become more noticeable on models like Qwen2 point 5-7B <ref:2603.23985#pg4>.

Tom: That’s fair, Meng. So it’s a promising path for efficient model compression that keeps task awareness without demanding a full retraining cycle for every single use case <ref:2603.23985#pg1>.

Conclusion: Tom: So, we're wrapping up our look at this paper, "Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score." Basically, they’ve developed a way to prune large language models without needing to retrain them for every new task.

Jane: Right. So it boils down to using task importance scores—how important different parts of the model are for specific tasks—and then merging those scores into one single global mask so you can prune the model once.

Lu: What’s interesting about that merge is how they use majority voting to decide which dimensions are safe to remove globally, even if some individual tasks disagree on a dimension. It finds that what multiple tasks independently flag as unimportant is likely good for everyone.

Meng: From a deployment standpoint, it means we don't have to maintain separate pruned versions for every single application the model might face; we get one compressed model that works across the board with modest profiling costs. That’s something I can actually see being useful on a production server.

Tom: So, what does this mean for us right now? It shows that you can get task awareness in pruning without needing massive training cycles or keeping a whole family of pruned models ready to go.

Jane: Exactly. It suggests that using small samples per task to guide this process is a manageable way to find important dimensions across different uses, which is much more efficient than traditional methods.

Lu: The authors show this works across several Gemma models and other architectures, which validates the idea that this isn't just a specific trick for one model.

Meng: But they did flag some limits; they mentioned that performance drops noticeably as you push the sparsity ratio higher, especially on larger models like Qwen2 point five-7B. That tells us we still need to figure out a good schedule for when to prune aggressively.

Tom: So it’s a solid method for getting task-aware compression with much less overhead than what we usually see in these papers. Next up, we're looking at how they actually implemented this masking operation on the model layers themselves.

Jimyung Hong Jaehyung Kim

Yonsei University

cs.LG

Submitted: 2026-03-25

Updated: 2026-10-18

Comments: 18 pages. Code available at https://github.com/Jimmy145123/DIET

Code: https://github.com/Jimmy145123/DIET

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: DIET is a training-free dimension-wise global pruning method that merges task-wise importance scores to produce a single global mask applicable across all tasks, achieving significant

Key concepts

Task-wise Importance Scoring: This involves measuring how important each dimension is for a specific task by profiling the LLM on small sets of samples from that task. It calculates an importance score for each dimension based on activation magnitudes across transformer blocks, tokens, and samples to identify dimensions minimally contributing to a particular task.
Majority Voting Aggregation: DIET combines the individual importance scores from different tasks into one global decision using majority voting. If multiple tasks independently flag a dimension as unimportant, it is likely safe to prune globally. This aggregation creates a unified mask that applies the same pruning decision across all layers for any given task.
Global Pruning Mask: This is the final binary selector derived from the task-specific assessments. It dictates which dimensions should be pruned uniformly across all sublayers of the LLM. This single mask allows a model to be compressed for deployment while retaining task-aware capabilities without needing separate versions for each use case.

Terminology

Pruning — 6× in this paper · explained in 4 episodes
Sparsity — 2× in this paper · explained in 5 episodes
Accuracy — 1× in this paper · explained in 1 episode
Adaptability — 1× in this paper
Benchmark — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 9 episodes

Summary

The gist: DIET is a training-free dimension-wise global pruning method that merges task-wise importance scores to produce a single global mask applicable across all tasks, achieving significant accuracy improvements over state-of-the-art structured pruning methods.

How it works

DIET derives a global pruning mask through a twophase process involving task importance scoring and aggregation First, DIET identifies the dimensions that contribute minimally to each task. Specifically, for each task t, it profiles the pretrained LLM fθ on N samples from task t to measure activation magnitudes at the output of the MLP layer in each transformer block. It then computes a per-dimension importance score a(t)k by averaging absolute activations across all blocks, tokens, and samples. Given a target sparsity s ∈ [0, 1], the dimensions with the smallest a(t)k are identified as pruning candidates to produce a binary selector z(t).

Merging into global pruning mask

The core innovation of DIET is aggregating these task-specific assessments to create a unified decision. To this end, it employs a simple idea of majority voting where dimensions that multiple tasks independently flag as unimportant are likely safe to remove globally without significantly harming individual tasks. This aggregation is performed by stacking taskwise selectors into a matrix Z = [z(t)]t∈T ∈ 1d×T and counting votes per dimension ck = P t∈T z(t)k. The indices for global pruning O are then selected as the top floor of dimensions with the highest vote counts, defined by O = arg top⌊s·d⌋ ck d k=1. This results in a single global pruning mask m ∈ 1d by setting mk = 1 if k ∈ O.

From Global Masking to Hard Pruning

The derived global mask is applied uniformly across all parametrized sublayers. For layers with input dimension d, input-dimension masking is applied using the operator y = W Pz x + b, where Pz = diag(z) (1). For layers with output dimension d, output-dimension masking uses the operator y˜ = P˜ z (W x + b), where P˜ z = diag(z) (2). For real-time efficiency evaluations, a hard pruning setting is considered where masked dimensions are physically removed and the hidden dimension is compacted from d to d′ = X Σ d j=1 u j, where u = 1 − m denotes the keep selector. To compensate for the variance reduction caused by reducing input dimensionality, a weight rescaling factor α = r / (d d′) is applied to linear sublayers whose input dimension is reduced, resulting in Wscaled = α · Wsliced.

Experimental Results and Advantages

Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET. For example, at 10% sparsity on Gemma-2 2B, DIET achieves near 70.6% average accuracy, compared to previous state-of-the-art structured pruning methods. Across both model sizes and all sparsity levels, DIET significantly outperforms a recent structured pruning method and a router-based pruning approach. Furthermore, DIET exhibits a clear advantage over task-wise depth pruning algorithms, achieving more than 20% accuracy increase on zero-shot commonsense reasoning tasks. The analysis of cross-task agreement via voting reveals that dimensions receiving 7 or 6 votes account for approximately 10% of total dimensions, enabling the achievement of target sparsity with strong agreement.

Generalization and Robustness

DIET's generalization to various LLMs is validated by conducting experiments on Qwen2.5-7B and Phi-4-Mini-Reasoning. Under a moderate pruning budget of 10% sparsity, DIET preserves most of the original accuracy across both backbones, demonstrating strong cross-architecture generalization without any retraining. However, as the sparsity ratio increases, performance degradation becomes more pronounced approximately 33% on Qwen2.5-7B and 21% on Phi-4-Mini-Reasoning.

Conclusion

DIET is a training-free dimension-wise global pruning framework that merges task-wise importance estimates into a single global mask. By profiling MLP activations on small per-task samples and aggregating through majority voting, DIET systematically removes residual dimensions that multiple tasks deem uninformative. Across seven zero-shot benchmarks on Gemma-2 2B and 9B, DIET consistently outperforms previous stateof-the-art structured pruning baselines at matched sparsity levels, achieving improvements of up to 21% at 10% sparsity ratio. In addition, DIET achieves consistent reductions in FLOPs and peak memory under hard pruning, leading to measurable latency improvements without degrading zero-shot performance. This results indicate that a simple, router-free, activation-driven approach can deliver robust task-aware compression with modest profiling costs.

Limitation and future work

Our evaluation focuses on seven English-centric zero-shot benchmarks; broader coverage across multilingual domains, safety-critical tasks, long-context scenarios, and tool-use applications remains unexplored. At high sparsity levels on non-Gemma architectures, we observe notable accuracy degradation, suggesting the need for adaptive sparsity schedules, per-layer allocation strategies, or lightweight postpruning recovery techniques. Also, task-weighted or distribution-aware voting mechanisms may improve robustness, while smarter calibration policies could enhance sample efficiency.

Broader Impact and Ethical Implications

By reducing computational and memory requirements through a single global mask, DIET offers practical benefits for model deployment and environmental sustainability. Unlike task-specific pruning methods that require maintaining multiple pruned variants, DIET’s global mask produces a single task-aware model capable of handling diverse tasks effectively. This consolidation simplifies deployment pipelines, reduces storage overhead, and lowers both energy consumption during inference and the carbon footprint associated with model distribution. By eliminating the need for multiple model variants, DIET makes large language models more accessible to resource-constrained settings while maintaining task-aware capabilities. However, pruning alters internal representations and may shift model behavior in fairness, safety, and robustness. Practitioners should audit pruned models for bias and distribution shift, rerun safety evaluations, and transparently document sparsity levels and calibration procedures.

Acknowledgments

Jimyung Jaehyung are affiliated with the Department of Artificial Intelligence at Yonsei University. This research was supported in part by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS2020-II201361, Artificial Intelligence Graduate School Program (Yonsei University); No. RS-2025- 25442405, Development of a Self-Learning World Model-Based AGI System for Hyperspectral Imaging).

References

Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns.

Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. 2023. Lorashear: Efficient large language model structured pruning and knowledge recovery.

Improvements for AI systems

Activation-based Profiling for Task-Aware Pruning: The system can generate a per-dimension importance vector by averaging absolute activations across all blocks, capturing cross-layer consistency to quantify each dimension's contribution to a specific task.
Majority Voting for Global Mask Construction: DIET aggregates task-wise selectors using majority voting to create a single global pruning mask, identifying dimensions that multiple tasks independently flag as unimportant are likely safe to remove globally without significantly harming individual tasks.
Task-Aware Generalization: The resulting system yields a single global mask which is easy to apply that is task-agnostic, meaning the pruned model serves all tasks without per-task customization, unlike prior methods that require task-specific training or routing.
Robustness to Profiling Cost: The method achieves near-optimal performance with a modest profiling budget, as demonstrated by reaching near saturation by sampling only 100 samples per task, indicating an effective balance between profiling cost and zero-shot benchmark accuracy.
Real-Time Efficiency Gains via Hard Pruning: When combined with variance correction, the system can achieve consistent reductions in FLOPs and peak memory during hard pruning, leading to measurable latency improvements while maintaining accuracy comparable to the zero-masked model.
Preservation of Instruction Following: For instruction-following tasks (IFEval), DIET shows that it maintains competitive or improved performance, preserving instruction-following behavior more effectively in larger models, which is critical for complex reasoning applications.