Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization".

Jane: Here's the gist of the paper. The gist Large Language Models (LLMs) have achieved widespread adoption due to their strong reasoning and queryresponse capabilities, but deploying them in embedded and edge computing environments remains challenging because strict latency, memory, and resource constraints limit efficient execution <ref:2607.22583#pg2>.

Tom: The proposed hardware-aware two-stage structured pruning framework for LLM compression outperforms existing methods on downstream commonsense reasoning tasks on average, achieves lower perplexity on WikiText-2, C4, and FineWeb-Edu, and reduces 100-token inference latency to 73.8512 s on an NVIDIA A100 GPU and 843 s on the evaluated Jetson Nano edge platform <ref:2607.22583#pg7>.

Jane: How it works The proposed methodology employs a global-to-local pruning framework that separates macro-level architecture search from fine-grained structural refinement <ref:2607.22583#pg2>.

Tom: This framework is executed in two distinct stages, as summarized in Algorithm 1 <ref:2607.22583#pg4>.

Title and authors: Tom: Now we move into the title of this paper, "Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization." It sounds a bit heavy, but it really captures the essence of what they are doing.

Jane: It’s a lot of technical terms packed in there, but basically they are proposing a systematic method for pruning large language models that doesn't just focus on making them smaller or improving their accuracy in isolation.

Lu: They aren't just looking at one metric; they are jointly optimizing performance preservation and latency reduction across multiple objectives >

Meng: So it’s about finding a way to simultaneously reduce the model size and improve how quickly it runs on hardware, which is a common challenge when moving AI from the lab to real applications.

Lalam: The title suggests they are focusing on structure; they are pruning specific units like attention heads and MLP layers rather than just randomly removing weights >

Tom: Exactly, because random pruning often leads to unpredictable performance drops, but by structuring it around MHA and FFN blocks, they can manage that design space more effectively >

Jane: And the authors are proposing a two-stage framework specifically to solve this complex design space problem by breaking it down into manageable steps.

Lu: They break the search down into identifying coarse candidates first in Stage one, which finds a Pareto set of layer-wise pruning configurations based on performance and latency >

Meng: That sounds like they are trying to avoid getting lost in an unmanageable search space right away by finding good starting points first.

Tom: Right, and then Stage two takes those promising candidates from the first stage and applies parallel Bayesian optimization to allocate the remaining sparsity budget for the final refinement >

Jane: So Stage two is about fine-tuning those already promising architectures to maximize performance retention while minimizing latency under specific constraints >

The paper's summary: Tom: To summarize the paper, "Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization," they present a two-stage framework for structured LLM compression that addresses the limitations of existing methods.

Jane: They specifically address how existing approaches mainly focus on accuracy preservation or parameter reduction while treating latency mostly as a post-pruning metric, which is what this paper sets out to fix.

Lu: This paper proposes a two-stage, multi-objective structured pruning framework that identifies a Pareto set of layer-wise pruning configurations in the first stage by jointly optimizing performance preservation and latency reduction >

Meng: So the core contribution is moving away from a single candidate search to this staged approach, which makes the search more efficient and stable >

Lalam: They are using specific structural units like Multi-Head Attention groups and Feed-Forward Network neuron groups as their units for pruning, like G MHA l,h or G MLP l,u >

Tom: And in Stage one, they search for a decision vector x that minimizes performance loss O1(x) and structural complexity O2(x) >

Jane: So the goal there is to find configurations where both accuracy degradation and model size are minimized simultaneously >

Lu: That search space Θ encompasses all possible pruning configurations across the entire model architecture, defined by binary variables for retaining or pruning MHA and FFN modules in each layer l >

Meng: I'm trying to picture how they define the performance loss O1(x) mathematically—is it just the KL divergence between original and pruned models?

Tom: Yes, the KL divergence over the vocabulary token space V is formulated as DKL(P∥Q) = X v∈V P(v) log P(v) Q(v), which is what they use for O1 >

Jane: And in Stage one, to execute that search, layer pruning is done by replacing the forward pass of targeted sub-blocks with an identity mapping to bypass pruned layers via skip connections >

Lu: That mechanism helps ensure the search can proceed through the architecture even when parts are removed >

The paper's improvements: Tom: So what are the specific improvements they suggest? They aren't just about one thing, but about improving how the pruning is done structurally.

Jane: The main improvement is proposing this two-stage, multi-objective structured pruning framework that jointly optimizing performance preservation and latency reduction >

Lu: This means Stage one identifies a Pareto set of layer-wise configurations by jointly optimizing performance preservation and latency reduction >

Meng: And then Stage two applies parallel Bayesian optimization over the returned Pareto-front solutions to identify a pruning configuration that maximizes performance retention while minimizing latency >

Tom: That second stage is where they use importance estimators to guide the allocation search, which is different from just picking one candidate from the first stage, which I think is crucial for finding the optimal trade-off >

Jane: They also show that a critical difference analysis shows that their approach using a mixed calibration set achieves the best average rank compared to other methods >

Lu: And they confirmed that a reversed stage-wise setting yields lower downstream accuracy and higher latency than this proposed H2SP order >

Meng: So it’s not just about the sequence; the specific order of operations matters, and they found H2SP is better because it addresses latency constraints better early on >

Tom: And they also showed that for objective combinations, KL/Params is preferable when accuracy preservation is prioritized over KL/Latency when you have very strict latency limits >

Jane: That gives practitioners a clear roadmap on how to configure the pruning objectives based on what their application actually demands >

Conclusion: Tom: Wrapping up this paper, the "Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization" shows that combining Pareto-based coarse pruning with latency-aware Bayesian sparsity allocation is an effective way to get an accuracy-efficiency trade-off for structured LLM compression >

Jane: The results confirm they improved average commonsense reasoning performance over existing pruning and sparsity-allocation methods, showing competitive or lower perplexity on WikiText-two C4, and FineWeb-Edu > <ref:2607.22583#pg1>

Lu: They found that the proposed approach achieves reduced one hundred-token inference latency on both NVIDIA A100 and Jetson edge hardware >

Meng: From a production standpoint, this means you can deploy models with better performance characteristics for real deployment where you have to worry about runtime constraints on devices like an NVIDIA A100 or the Jetson Nano >

Lalam: The paper emphasizes that combining Pareto-based coarse pruning with latency-aware Bayesian sparsity allocation provides a strong accuracy-efficiency trade-off for structured LLM compression >

Tom: This work gives us a solid method for optimizing how we prune these models to get better results on both performance and speed metrics >

Jane: It’s a detailed framework that shows how to structure the compression process to ensure the search is both efficient and stable across all those objectives >