Activation Sensitivity as a Unifying Principle for Post-Training Quantization

arXiv:2601.11663 · cs.LG, cs.AI · Submitted 2026-01-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Activation Sensitivity as a Unifying Principle for Post-Training Quantization".

Jane: Here's the gist of the paper. The gist: This work presents a unified theoretical framework for post-training quantization by formalizing activation sensitivity as the expected impact of channel-wise perturbations on the loss, revealing that existing methods are complementary approximations of this fundamental quantity.

Tom: Post-training quantization (PTQ) methods for large language models rely on heuristics that implicitly estimate which weight channels matter most to model behavior Two dominant paradigms have emerged: activation-aware methods such as AWQ prioritize channels with large activation magnitudes, while second-order methods such as GPTQ allocate quantization error according to input covariance structure.

Jane: Despite their empirical success, these approaches remain conceptually fragmented, and it is unclear what underlying quantity they are approximating.

Tom: The Unified Framework of Activation Sensitivity The central object underlying PTQ is neither activation magnitude nor input covariance alone, but rather activation sensitivity: the expected impact of channel-wise perturbations on the loss.

Title and authors: Jane: Now let’s look at the actual summary of "Activation Sensitivity as a Unifying Principle for Post-Training Quantization," because it lays out exactly what this framework is trying to achieve beyond just naming things.

Tom: It says the central object underlying all these post-training quantization methods isn't just activation magnitude or input covariance; it’s activation sensitivity, defined as the expected impact of channel-wise perturbations on the loss.

Lu: And how they formalize this is through a first-order Taylor expansion of the loss function to show that this sensitivity emerges naturally as the squared norm of gradient-weighted activations.

Meng: That’s what makes it powerful because it means this measure captures both how strongly a channel is used and how errors in that channel propagate downstream.

Jane: It’s not just about measuring usage frequency, but about measuring whether perturbations in that channel actually affect the final loss, which is what the paper calls the effective coefficient governing how perturbations to channel j influence the loss.

Tom: So they derive this from an output perturbation delta Y which is related to a weight perturbation delta W with a gradient G of the loss with respect to layer output.

Lu: The derivation shows that by averaging over calibration samples, you get the quantity E delta Y squared proportional to delta W squared times E X squared, which recovers the activation-magnitude criterion used by activation-aware methods.

Meng: But then they specialize that first-order expansion for channel-wise perturbations to get the loss-aware sensitivity, defined as alpha j ≡ E G T X:,j squared.

Jane: That alpha j is what captures how strongly a channel is used *and* how errors in that channel propagate downstream through the computation, which is where the real value of this paper lies.

Tom: So existing PTQ methods are approximations of this sensitivity, but they lack the explicit link to loss-derived structure that this paper provides.

Lu: They succeed or fail depending on things like architectural details and activation distributions without a clear explanation for why.

Meng: That’s a problem because we can’t debug the failure mode if we don't know what structural property we missed.

Jane: Exactly, it shifts the focus from just picking a metric to understanding the mathematical properties of the loss function that determine channel importance.

The paper's summary: Tom: So, looking at what they suggest as improvements, the first one is designing quantization based on this activation sensitivity metric directly to capture downstream error propagation explicitly.

Lu: That’s the main goal: moving beyond just activation magnitude prioritization by using a measure that simultaneously considers both usage and loss impact.

Meng: And the second improvement they propose is to handle cross-layer sensitivity dynamically, which accounts for how errors in early layers change later layer activations and gradients.

Jane: That’s crucial because it addresses the issue where local objectives fail due to unmodeled cross-layer feedback loops that we see in practice.

Tom: And the third improvement involves making the system robust against distribution shift by using adaptive or iterative approaches that update sensitivity estimates when quantization changes things.

Lu: This means sensitivity isn't a static property of the pretrained model anymore; it can evolve, which is a big deal for accuracy at inference time.

Meng: If we can make the sensitivity estimate dynamic, it should give us better accuracy when moving from calibration data to actual inference conditions where things have shifted.

Jane: And finally, they suggest selecting between different approximation regimes based on computational limits and fidelity by explicitly choosing assumptions like gradient-based saliency or Fisher-based metrics.

Tom: So the paper is giving us a toolset here: a principled measure, dynamic cross-layer handling, and flexibility in picking the right mathematical assumption for our specific deployment constraints.

Lu: It’s about understanding when existing heuristics are valid versus when they fail because they neglect downstream curvature information.

The paper's improvements: Tom: So to wrap up on "Activation Sensitivity as a Unifying Principle for Post-Training Quantization," the core message is that we need to move from using fragmented heuristics to using a unified, principled measure based on activation sensitivity.

Jane: This sensitivity, formalized as the squared norm of gradient-weighted activations, provides a principled way to quantify channel importance by linking usage frequency directly to its impact on the loss.

Lu: The main implication is that we gain a unifying lens for understanding why different PTQ methods work or fail based on architectural details and activation distributions.

Meng: For us in production, this means we can design quantization strategies that are more robust against distribution shifts by incorporating dynamic updates to sensitivity estimates during the deployment phase.

Tom: And the practical flexibility is choosing between different approximation regimes to match our computational budget with the fidelity we actually need at inference time.

Jane: This paper gives us a clear direction: instead of designing increasingly elaborate reconstruction losses, we should focus on understanding when approximate sensitivity is informative rather than just designing more complex ones.

Lu: It’s about building a framework that understands the structure of the loss function's response to channel perturbations across layers.

Conclusion: Tom: So we’ve talked about how this paper frames post-training quantization not as just picking activation magnitude or input covariance, but through a unified concept called activation sensitivity.

Jane: Exactly, Tom. The core idea is that sensitivity is actually the expected impact of channel perturbations on the loss function.

Lu: And they formalize that by using a first-order Taylor expansion around the loss surface to get this metric, which they call the squared norm of gradient-weighted activations.

Meng: For us in production, what matters is that this gives us a better way to predict which channels are most important for accuracy, not just which ones have big activations.

Lalam: I see it as a way to look at the model’s internal structure more deeply than just looking at input statistics.

Tom: Right. And the paper shows how existing methods like GPTQ or AWQ are actually approximations of this sensitivity, which is useful because it explains why they behave differently sometimes.

Jane: It also points out the limitations, especially around cross-layer effects and calibration mismatch, which is a huge deal for real-world deployment.

Lu: They flag that local layer objectives don't account for how errors accumulate across layers, which means the sensitivity isn't static in a way we thought it was.

Meng: That’s exactly what breaks things at scale. If early layer quantization shifts the activation distribution for later layers, our sensitivity estimate becomes totally wrong.

Tom: It also suggests that we can choose between different metrics based on what we need—whether we want something gradient-based or something more curvature-aware like Fisher metrics.

Jane: And it gives us a direction for future work: developing adaptive or iterative approaches that update these sensitivity estimates as the quantization changes things at inference time.

Lu: That’s where I think the real potential is, moving toward models that can sense and adjust their own importance based on the current quantization noise.

Lalam: I’m confident this framework gives us a solid foundation to build something more accurate without needing constant retraining for every quantization scheme.

Tom: So it clarifies what these methods are doing and where they stop working, which is valuable for engineers trying to choose the right tool for the job.

Jane: It’s a good way to understand why we might see layer-dependent behavior in these models.

Lu: This paper, "Activation Sensitivity as a Unifying Principle for Post-Training Quantization," gives us a much clearer lens on this whole quantization problem.

Meng: So we’ll be watching how this sensitivity concept plays out when we try to build our next generation of efficient models.

Bruce Changlong Xu

Stanford University

cs.LG, cs.AI

Submitted: 2026-01-15

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist: This work presents a unified theoretical framework for post-training quantization by formalizing activation sensitivity as the expected impact of channel-wise perturbations on the loss,

Key concepts

Activation Sensitivity: This is the core concept, defined as the expected impact of changing a specific weight channel on the final model loss. It moves beyond just looking at how much a channel is activated; it measures how much that channel's perturbation will actually change the model's performance during inference.
Gradient-Weighted Activations: This is a specific metric derived from activation sensitivity. It captures two things: how strongly a channel is used (activation magnitude) and how sensitive the loss is to errors in that channel (gradient influence). This provides a more nuanced measure of which channels are critical for the model's output.
Second-Order Methods: These methods, like GPTQ, approximate activation sensitivity by looking at input covariance structure. They try to estimate error propagation using statistical properties of the data rather than direct loss calculations. This is a simplified approach that ignores complex interactions between layers and activations.

Terminology

Quantization — 5× in this paper · explained in 15 episodes
Fisher Information — 1× in this paper · explained in 4 episodes
Post-Training Quantization (PTQ) — 1× in this paper · explained in 2 episodes
Pruning — 1× in this paper · explained in 4 episodes

Summary

The gist: This work presents a unified theoretical framework for post-training quantization by formalizing activation sensitivity as the expected impact of channel-wise perturbations on the loss, revealing that existing methods are complementary approximations of this fundamental quantity.

Introduction to Post-Training Quantization Paradigms

Post-training quantization (PTQ) methods for large language models rely on heuristics that implicitly estimate which weight channels matter most to model behavior Two dominant paradigms have emerged: activation-aware methods such as AWQ prioritize channels with large activation magnitudes, while second-order methods such as GPTQ allocate quantization error according to input covariance structure. Despite their empirical success, these approaches remain conceptually fragmented, and it is unclear what underlying quantity they are approximating.

The Unified Framework of Activation Sensitivity

The central object underlying PTQ is neither activation magnitude nor input covariance alone, but rather activation sensitivity: the expected impact of channel-wise perturbations on the loss. This is formalized by analyzing how small weight perturbations affect the loss through a first-order Taylor expansion. This derivation yields a simple and interpretable saliency metric—the squared norm of gradient-weighted activations—that captures both how strongly a channel is used and how errors in that channel propagate downstream.

Relationship to Existing Quantization Methods

Within this framework, AWQ can be interpreted as approximating sensitivity under the assumption of uniform downstream gradients, reducing sensitivity to activation magnitude. Second-order methods such as GPTQ emerge when sensitivity is approximated through unweighted input covariance, effectively marginalizing over activation structure. More sophisticated reconstruction-based methods, including block-wise objectives [3], can be viewed as partial attempts to account for interactions that layer-local approximations neglect.

Limitations of Layer-Local Proxy Objectives

Because sensitivity is defined with respect to the loss, it depends on downstream computation, calibration distributions, and cross-layer error accumulation—factors that cannot be fully captured by reconstruction objectives optimized independently per layer. This helps explain why PTQ methods often exhibit layer-dependent behavior. Furthermore, quantization error accumulates across layers, which violates the assumption that errors introduced earlier do not alter the sensitivity structure of later layers.

Design Space of Sensitivity Metrics

The design space of sensitivity metrics involves trade-offs between granularity and stability. First-order metrics, like gradient-norm saliency measures, directly reflect downstream loss behavior but are inherently local and ignore curvature. Second-order sensitivity metrics attempt to capture amplification by local curvature, with covariance-based objectives approximating the Hessian using layer input statistics. Fisher-based metrics offer an alternative approximation grounded in information geometry, interpolating between gradient-norm and covariance criteria.

Connections to Classical Second-Order Compression Theory

Classical pruning methods like Optimal Brain Damage (OBD) and Optimal Brain Surgeon (OBS) estimate parameter importance through second-order approximations of the loss under parameter deletion. A central distinction is that quantization introduces dense, structured perturbations across channels and layers, which may cause second-order approximations accurate for sparse perturbations to fail to capture the aggregate effect of dense quantization noise.

Open Challenges in Post-Training Quantization

Key open challenges include cross-layer sensitivity and error accumulation, where perturbations reshape downstream geometry. Calibration distribution mismatch is another issue, as sensitivity estimates derived from calibration data may fail to reflect true importance at inference time. Finally, the tension between static sensitivity estimates and the dynamic nature of quantization-induced distribution shifts suggests that adaptive or iterative approaches are a promising direction.

Conclusion on PTQ Understanding

The sensitivity-based framework clarifies what common objectives are doing and makes explicit where their approximations are brittle, such as cross-layer feedback and calibration mismatch. Progress depends on understanding when approximate sensitivity is informative rather than just designing increasingly elaborate reconstruction losses.

References

[1] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems

[2] Elias Frantar and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations

[3] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. 2021

[4] Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, 1990

[5] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, 1993

[6] Sheng Shen, Zhen Dong, Ji Ye, Lin Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. In AAAI Conference on Artificial Intelligence

[7] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ: Hessian aware quantization of neural networks with mixed-precision. 2019

[8] Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ-V2: Hessian aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems

[9] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition

[10] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. 2023

[11] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems

[12] Guangxuan Xiao, Ji Lin, Matthieu Seznec, Jeffrey Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning

[13] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. 2023

--- Page 1 ---

ACTIVATION SENSITIVITY AS A UNIFYING PRINCIPLE FOR

Post-training quantization (PTQ) methods for large language models rely on heuristics that implicitly estimate which weight channels matter most to model behavior. Two dominant paradigms have emerged: activation-aware methods such as AWQ prioritize channels with large activation magnitudes, while second-order methods such as GPTQ allocate quantization error according to input covariance structure.

Improvements for AI systems

Activation sensitivity-based quantization can be designed to explicitly capture downstream error propagation by using a metric derived from the squared norm of gradient-weighted activations. This allows for quantization decisions that prioritize channels based on both their usage magnitude and their impact on the final loss, moving beyond simple activation magnitude prioritization.
The system can dynamically estimate cross-layer sensitivity to mitigate error accumulation across layers by incorporating a framework that accounts for how quantization error introduced in early layers alters activation distributions and gradients in later layers. This prevents local layer-local objectives from failing due to unmodeled cross-layer feedback loops.
Quantization methods can be made more robust against distribution shift by developing adaptive or iterative approaches that update sensitivity estimates in response to quantization-induced changes. This addresses the limitation that sensitivity is not a static property of the pretrained model, allowing for better accuracy when moving from calibration data to actual inference conditions.
The system can select between different approximation regimes based on computational constraints and desired fidelity by explicitly choosing which assumptions are made, such as approximating sensitivity as gradient-based saliency or using Fisher-based metrics. This allows engineers to understand precisely when existing heuristics are valid and when they fail due to neglecting downstream curvature.

Abstract

Post-training quantization (PTQ) methods for large language models compress pretrained weights to low bit-widths without retraining by optimizing layer-local proxy objectives. Activation-aware scaling methods such as AWQ protect input channels with large activation norms, whereas second-order error-compensation methods such as GPTQ redistribute rounding error using the inverse of the unweighted input covariance matrix. We show that both paradigms -- along with diagonal Fisher quantization, block reconstruction, and Hadamard rotation methods -- are special cases of minimizing a single loss-derived quadratic form governed by gradient-weighted activation sensitivity. From a Gauss-Newton expansion of the task loss at a linear layer Y = X W, we prove that under isotropic output curvature the expected excess loss from weight perturbation ΔW reduces to 1 over 2 d out Tr(ΔW H D ΔW), where H D = 1 over n X D X weights each calibration token's outer product by its squared output-gradient norm D tt = | grad y t L| 2 squared. Within this framework, AWQ corresponds to a diagonal approximation of H D with uniform token weights (D = I n) optimized via channel scaling, while GPTQ corresponds to Cholesky error compensation on H D under D = I n. We derive closed-form optimal scaling exponents for diagonal sensitivity protection, formulate sensitivity-weighted Cholesky quantization (SAQ-GPTQ) on H D, and prove bounds on token-gradient heterogeneity bias, cross-layer error amplification across residual blocks, and finite-sample calibration concentration. Controlled numerical simulations confirm that under token-gradient heterogeneity, replacing X X with X D X preserves >99% top-channel saliency recall and reduces INT3/INT4 excess loss by roughly 3 times over unweighted GPTQ.