InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization

arXiv:2605.26175 · cs.LG, cs.AI · Submitted 2026-05-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization".

Jane: Here's the gist of the paper. The gist Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment, and INFOQUANT proposes a train-free method to shape activations into more quantization-friendly distributions by analyzing information-theoretic principles.

Tom: Post-training quantization (PTQ) methods often fail because they do not specify what transformed activation distribution is actually easy to discretize.

Jane: From an information-theoretic perspective, the analysis shows that lower quantization error is associated with two complementary properties: a smaller numerical range and greater dispersion within that range.

Tom: This reframes the role of activation transformation, suggesting that a PTQ method should explicitly shape activations into distributions that are easier for a low-bit quantizer to preserve.

Title and authors: Tom: So we're looking at this paper called "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization," and it tackles a big problem in deploying large language models efficiently on hardware.

Jane: Right, so the core issue is that when you try to compress these models using low-bit quantization, the way we usually do it doesn't actually tell the quantizer what distribution to expect.

Lu: Exactly. Existing methods might suppress peaks or minimize reconstruction loss, but they don't specify a target distribution that is easy for a low-bit quantizer to handle, which is why activations can still have large quantization errors even if they look smoother.

Meng: From my side, I worry about deployment cost. If we can't predict the distribution well enough, we end up with huge accuracy drops at scale where the hardware constraints are tight.

Lalam: I looked at the paper, and it recasts this problem by looking at how quantization error depends on the activation distribution after a transformation; it shows that lower quantization error comes from two things: a smaller numerical range and greater dispersion within that range.

Tom: So they aren't just trying to smooth out the peaks; they are trying to shape the distribution into something where the quantizer can actually preserve it well, which is what this paper calls "quantizer-facing distribution design."

Jane: That makes sense. If we can make the range smaller but keep the values spread out within that smaller range, it should be much easier for a low-bit quantizer to map everything correctly.

Lu: The main part of their method is something called Peak Suppression Orthogonal Transformation or PSOT; this learns an orthogonal transformation by directly penalizing tokens that are dominated by peaks.

Tom: I like that they are learning the transformation, not just using a fixed rotation like Hadamard rotation; that suggests it adapts to the specific model structure.

Meng: What about the actual performance gains? The paper claims this method produces a more quantization-friendly distribution with a narrower numerical range and larger normalized dispersion.

Jane: They show this by comparing the original activations, a standard Hadamard rotation, and then the one learned by PSOT; it clearly shows PSOT achieves that better distribution shape.

Lu: They use their notation where lambda is defined as lambda = /b n, and they say a smaller lambda indicates a lower normalized quantization-error bound, which is related to the range-normalized quantization step size s.

Title and authors: Tom: That's a concrete measure of success; instead of just saying "it's better," they give us this lambda metric to track how much error we are reducing.

Jane: And they connect that back to the idea that lower error is associated with this two-part property: smaller range and greater dispersion, which reframes how we should think about activation transformation in post-training quantization.

Lalam: The paper introduces three main components for INFOQUANT: PSOT learns the orthogonal transformation; Adaptive Outlier-Token Selection or ASOT emphasizes informative calibration tokens by selecting reliable outliers during rotation optimization; and Learnable Activation Clipping or LAC refines the final quantization interval using parameters alpha and beta.

Tom: So it's not just one step, it’s a whole system where PSOT shapes the distribution, ASOT picks the best data points to focus on during that shaping, and LAC does the final fine-tuning of how we clip things.

Lu: The component ablation study shows that replacing a fixed Hadamard rotation with their learned PSOT rotation yields the largest single change in perplexity, which suggests that learning where activation energy should be redistributed is the most important part.

Meng: That's good to see; if the core gain comes from learning the transformation itself, then we can focus our engineering effort there instead of trying to tweak fixed rotations.

Jane: And they also talked about how ASOT improves optimization robustness by selecting reliable outlier tokens based on sample-dependent inconsistencies, which means the learned rotation generalizes better across different inputs rather than just overfitting to fixed positions in the sequence.

Tom: It sounds like the flow is: first reshape the distribution, then reconstruct weights, and then refine which calibration tokens and clipping ranges get our attention. That ordering of gains is interesting.

Lalam: They also found that increasing the number of blocks in block-diagonal orthogonal matrices causes a decline in quantization performance, which gives us some insight into how complex these transformations can get before they start hurting the result.

Jane: So, to summarize this paper on "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization," it proposes a train-free post-training quantization method that learns orthogonal transformations to create distributions that are naturally easier for low-bit quantizers by focusing on a smaller numerical range and greater dispersion.

Tom: And the empirical results show real gains, like preserving ninety-seven percent of floating-point accuracy on average under W4A4KV4 settings, and reducing the LLaMA-two 13B performance gap by forty-two percent.

Lu: They also found that for Qwen-two point five models, INFOQUANT maintains superior performance on both the 14B and 32B models when using four-four-four quantization.

Title and authors: Meng: From a deployment standpoint, they show significant hardware efficiency gains, achieving a prefill speedup of two point nine one times and a memory saving factor of two point five nine times for LLaMA-three 8B models in the W4A4KV4 setting.

Jane: The implication here is that this method makes quantization much more practical for consumer hardware, allowing models like the 70B model to be quantized using only about twenty-four gigabytes of GPU memory.

Tom: It suggests that activation quantization should be evaluated by both reducing outliers and effectively using the available low-bit levels, which is a useful piece of advice for anyone working on this area.

Lu: The authors themselves flag a limitation: their findings are interpreted within the scope of the evaluated settings, and they suggest extending this analysis beyond bell-shaped activation assumptions for broader distributional regimes.

Meng: I agree with Lu; it's important to remember that calibration conditions can change significantly, so we can't just assume these exact results will hold everywhere.

Jane: So, while INFOQUANT is very effective under the tested conditions, the next step seems to be looking at what happens when activation patterns are more complex than bell-shaped ones.

Tom: It’s a solid method for now because it's train-free and gives these measurable improvements in accuracy and speed.

Lu: The way they approached this—recasting activation transformation as quantizer-facing distribution design—is the big idea here, moving away from just trying to suppress peaks or minimize simple reconstruction errors.

Meng: It’s about designing the input for the quantizer, which is a much more proactive approach than reacting to what happens after training.

Lalam: The paper "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization" suggests that we should prioritize shaping activations into distributions that are inherently easier to preserve using low-bit uniform quantizers.

Tom: That's the gist of it—if you understand what the quantizer needs, you can make the activation fit it better.

Jane: So for listeners, this means that when we think about making these massive language models run on phones or standard GPUs, we need to look at how we are preparing those activations before quantization.

Lu: We need to keep an eye on extending this analysis because assuming bell-shaped activations isn't always accurate across all model architectures and training regimes.

Meng: For deployment engineers, it means we can target specific hardware constraints knowing that a well-shaped distribution gives us better memory savings without sacrificing too much accuracy.

Jane: So, we see a method that improves accuracy while providing better efficiency metrics for deploying these large models today.

The paper's summary: Tom: So, to recap, InfoQuant is this train-free method that learns how to shape activation distributions specifically so they play nice with low-bit quantizers by managing both the numerical range and the dispersion of those activations.

Jane: Right, it’s not just about making things smooth; it’s about making them look like something a low-bit system can actually handle without losing too much information.

Lu: The core idea they found is that lower quantization error comes from two things happening at once: you want a smaller numerical range for the numbers, but you also need those numbers to be spread out nicely within that smaller range.

Meng: So, if we can get the distribution shaped like a tight ball instead of a stretched oval, it should make mapping to low bits much more predictable.

Tom: Exactly. Their method does this through three parts: they have Peak Suppression Orthogonal Transformation, PSOT, which learns the transformation by penalizing tokens that are just dominated by big peaks.

Jane: And PSOT is what actually does the heavy lifting of reshaping those activations into that ideal distribution shape we talked about.

Lu: Then you have Adaptive Outlier-Token Selection or ASOT, which picks reliable outlier tokens to focus on during the rotation optimization phase. That makes the learned transformation more robust across different inputs.

Tom: And finally, Learnable Activation Clipping, LAC, which uses learnable parameters alpha and beta to fine-tune exactly where we clip those activations so they match the full precision output perfectly again.

Jane: So it’s a three-step process: shape the distribution with PSOT, focus the learning on good tokens with ASOT, and then fine-tune the clipping points with LAC.

Meng: From what I see for deployment, this whole framework is designed to be train-free, which is huge because we don't have to run massive retraining cycles just to get better quantization performance.

Tom: It’s train-free, and they show some real numbers. They said under the W4A4KV4 setting—which is a pretty standard way of looking at memory usage—they preserve ninety-seven percent of floating-point accuracy on average for LLaMA models.

Lu: And they also showed a reduction in the performance gap between different model sizes, specifically cutting that gap by forty-two percent for the LLaMA-two 13B model compared to what came before.

Jane: That’s a big deal because it means we can deploy these large models on consumer hardware without having to sacrifice too much of their intelligence just because of the quantization step.

Tom: They even showed some impressive deployment efficiency gains, like a prefill speedup of two point nine one times and a memory saving factor of two point five nine times for LLaMA-three 8B models in that W4A4KV4 setting.

Meng: A memory saving factor of two point five nine times is significant; that means we can fit much larger models onto the same GPU with less constraint on VRAM.

Lu: The ablation study pointed out something key: replacing a fixed rotation, like the Hadamard rotation, with their learned PSOT rotation gave them the biggest jump in perplexity reduction.

Jane: So it really highlights that learning where to redistribute activation energy is more important than just trying a few different fixed shapes.

Tom: And Lalam confirmed that they are pretty sure about this direction because it aligns with what we see in the activation patterns of models like Qwen-two point five under four-four-four quantization, showing superior performance on both the 14B and 32B versions.

Meng: It’s a win for production because it’s empirically validated across several model families, not just one isolated test case.

Jane: But we have to remember what they said at the end: this analysis is really scoped to the settings they tested, so we can't assume these exact results will hold if you change your calibration conditions a lot.

Lu: That’s fair; it’s a warning that this method needs careful evaluation on different model architectures and training regimes if you want to use it broadly.

Tom: So, the paper is offering a concrete mechanism—PSOT, ASOT, LAC—that tackles the hard problem of distribution shaping for low-bit quantization without needing any extra training time.

Jane: And it suggests that for LLM deployment, we should look at how we prepare activations before quantization by focusing on these distributional properties.

Lu: What’s next is figuring out if this concept of shaping activations beyond the bell-shaped assumptions they used here can be extended to a wider range of activation patterns.

Meng: For my team, it means we can start prototyping with this structure to see what real-world gains we get before we commit to a full deployment pipeline overhaul.

The paper's improvements: Tom: So, we're looking at how they actually improve things compared to just having some general activation transformation idea.

Jane: Right, they show that their three components—PSOT, ASOT, and LAC—don't work in isolation; they build on each other for a much better final result.

Lu: The main improvement is that PSOT learns the orthogonal transformation directly by punishing tokens that are dominated by peaks; this specifically targets the distribution shape we talked about earlier.

Tom: And that learned transformation is what makes the biggest difference, because replacing a fixed rotation with their learned one gave them the largest single change in perplexity, which tells us learning the energy redistribution is critical.

Jane: That means they aren't just guessing a good rotation; they are letting the model figure out exactly how to best spread those high-energy activations so a low-bit system can handle them better.

Meng: From a production standpoint, this learnable component is what makes it robust enough to work across different inputs rather than just overfitting to some fixed sequence positions.

Lu: Then you have ASOT, which selects reliable outlier tokens during the rotation optimization; this ensures the learned transformation generalizes well across different data samples.

Tom: And LAC, Learnable Activation Clipping, uses parameters alpha and beta to fine-tune that final quantization interval so it matches the full precision block output perfectly.

Jane: So they’re not just transforming activations; they are also tuning exactly where to cut off the precision for each block based on what the system actually needs.

Meng: That matching step is important because it leads to a more precise final quantization step size, which helps keep that accuracy high when we move things into deployment.

Lu: The ablation study also showed that increasing the number of blocks in those block-diagonal orthogonal matrices actually hurt performance, so it gives us a constraint on how complex this transformation can get before it starts degrading the result.

Tom: It sounds like they found a good balance: PSOT for the big shape change, ASOT for robustness, and LAC for precision tuning.

Jane: And empirically, this system is showing excellent results; they’re preserving ninety-seven percent of floating-point accuracy on average under W4A4KV4 settings.

Meng: That level of accuracy preservation is exactly what we need when we're trying to get models running efficiently on hardware where every bit counts.

Lu: The implication for the world is that we can move toward a more proactive approach to activation quantization, designing the input distribution to be quantizer-friendly rather than just reacting after training.

Tom: And they are also pushing beyond the initial assumptions, suggesting we need to extend this analysis beyond just bell-shaped activations to cover a wider variety of model behavior.

Jane: So while this method works great for the settings tested, the authors themselves flag that if you change your calibration conditions substantially, these exact performance numbers might shift.

Lu: That’s a necessary caveat; it reminds us that research needs to be interpreted within the specific scope of the evaluation setup.

Meng: For someone deploying models today, this means we can target specific hardware constraints knowing that a well-shaped distribution gives us better memory savings without sacrificing too much accuracy in those measured scenarios.

Tom: So what’s next is figuring out how this concept of shaping activations beyond bell-shaped assumptions can be extended to cover a wider variety of activation patterns across different architectures.

Conclusion: Tom: So, to wrap things up on "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization," we’re talking about a train-free method that learns how to shape activations so they fit low-bit quantizers better by managing both their numerical range and dispersion.

Jane: Right, it’s this whole framework of PSOT, ASOT, and LAC working together to design the input distribution for maximum efficiency.

Lu: Basically, the paper shows that lower quantization error comes from making the numerical range smaller while keeping the values spread out within that smaller range.

Meng: That means when we deploy these models on hardware where memory is tight, we can use fewer bits and still keep things accurate.

Tom: And their empirical results are pretty solid; they're seeing ninety-seven percent of floating-point accuracy on average for LLaMA models under the W4A4KV4 setting.

Jane: That's a big number because it shows this approach actually delivers real performance gains over existing post-training quantization methods.

Lu: They also noted that the core gain comes from learning where to redistribute activation energy, which is why replacing a fixed rotation with their learned PSOT gave them the largest jump in perplexity reduction.

Meng: For deployment, it’s good that this is train-free; we don't have to run massive retraining cycles just to get better quantization performance.

Tom: And they showed some efficiency numbers too, like a prefill speedup of two point nine one times and a memory saving factor of two point five nine times for LLaMA-three 8B models in that W4A4KV4 setting.

Jane: That memory saving factor is significant; it means we can fit much larger models onto the same GPU with less constraint on VRAM.

Lu: The implication for the research community is that we should look at extending this analysis beyond just bell-shaped activations to cover a wider variety of activation patterns.

Meng: For my team, it means we can start prototyping with this structure to see what real-world gains we get before we commit to a full deployment pipeline overhaul.

Tom: So, the paper "InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization" suggests a proactive way to design activations specifically for low-bit quantization.

Jane: It’s about moving away from just reacting to activation patterns after training and instead designing them to be quantizer-friendly.

Lu: The challenge now is figuring out how this concept of shaping activations beyond bell-shaped assumptions can be extended to cover a wider variety of model behaviors.

Meng: From my side, I’m focused on seeing if we can integrate this PSOT structure into our existing quantization pipeline for our next production release.

Tom: It's a solid method for now because it's train-free and gives these measurable improvements in accuracy and speed under the tested settings.

Jane: And while they are confident in their results, we have to remember what they said: this analysis is really scoped to the settings they tested, so we can't assume these exact results will hold if you change your calibration conditions a lot.

Lu: That’s a necessary caveat; it reminds us that research needs to be interpreted within the specific scope of the evaluation setup.

Meng: For deployment engineers, this means we can target specific hardware constraints knowing that a well-shaped distribution gives us better memory savings without sacrificing too much accuracy in those measured scenarios.

Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He

School of Software Technology, Zhejiang University · Ant Group Research & Innovation Institute (China Mobile) · Alibaba Cloud Computing

cs.LG, cs.AI

Submitted: 2026-05-25

Updated: 2026-10-21

Comments: EMNLP2026 main

Code: https://github.com/LLIKKE/InfoQuant

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 80/100

The gist: The gist Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment, and INFOQUANT proposes a train-free method to shape activations into more

Key concepts

Peak Suppression Orthogonal Transformation (PSOT): This component learns an orthogonal transformation by specifically targeting tokens that have very high activation peaks. By penalizing these peaks, PSOT shapes the activation distribution to have a smaller numerical range and greater dispersion, making it easier for low-bit quantizers to accurately represent the data.
Adaptive Outlier-Token Selection (ASOT): ASOT identifies and selects 'reliable outlier tokens' during optimization. This ensures that the learned transformation is robust and generalizes well across different inputs rather than just fitting specific token positions, improving the overall performance of the quantization process.
Learnable Activation Clipping (LAC): LAC refines the final quantization step size using adjustable parameters alpha and beta. These parameters are optimized to ensure that the resulting quantized block output closely matches the original full-precision block output, achieving a more precise final interval selection.

Terminology

Quantization — 8× in this paper · explained in 15 episodes
Accuracy — 1× in this paper · explained in 1 episode
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
Peak Suppression Orthogonal Transformation — 1× in this paper
Post-Training Quantization (PTQ) — 1× in this paper · explained in 2 episodes
Reconstruction Error — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 6 episodes

Summary

The gist Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment, and INFOQUANT proposes a train-free method to shape activations into more quantization-friendly distributions by analyzing information-theoretic principles.

Motivation for the Approach

Post-training quantization (PTQ) methods often fail because they do not specify what transformed activation distribution is actually easy to discretize. From an information-theoretic perspective, the analysis shows that lower quantization error is associated with two complementary properties: a smaller numerical range and greater dispersion within that range. This reframes the role of activation transformation, suggesting that a PTQ method should explicitly shape activations into distributions that are easier for a low-bit quantizer to preserve.

INFOQUANT Components

INFOQUANT is a train-free PTQ framework consisting of three components:

Peak Suppression Orthogonal Transformation (PSOT) learns an orthogonal activation transformation by directly penalizing peak-dominated activation tokens. This objective promotes the two distributional properties identified by the KL analysis, which lowers the clipping scale and increases the max-normalized dispersion bn.
Adaptive Outlier-Token Selection (ASOT) emphasizes informative calibration tokens by selecting reliable outlier tokens and reweighting them during rotation optimization.
Learnable Activation Clipping (LAC) refines the final quantization interval through learnable parameters α and β to match the quantized block output to the full-precision block output.

Theoretical Analysis of Quantization Error

The analysis uses a smoothed KL divergence as an analytical lens for distributional distortion caused by low-bit quantization. This decomposition exposes the key mechanism, showing that a transformation that makes activations easier to quantize should not only suppress extreme values but also reduce the normalized error induced by the finite set of quantization levels. The resulting bound is expressed as FP,B(κ) = κ 2M + τP (κ), where κ = 1/bn. This analysis turns the vague goal of “making activations smoother” into a concrete distributional target: reduce the clipped numerical range while keeping the normalized activation values well dispersed inside that range.

Empirical Results and Performance

INFOQUANT consistently outperforms prior PTQ and end-to-end training baselines across multiple LLM families. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. The method demonstrates strong empirical and practical gains, with INFOQUANT achieving a WikiText2 perplexity of 6.89 on LLaMA-2 7B under W4A4KV4.

Deployment Efficiency

The method is designed to be hardware-efficient, with INFOQUANT achieving a prefill speedup of 2.91× and a memory saving factor of 2.59× for LLaMA-3 8B models in the W4A4KV4 setting. Furthermore, INFOQUANT enables quantization of the 70B model using just 24GB of GPU memory, demonstrating compatibility with consumer-grade hardware.

Conclusion

The work suggests that activation quantization should be evaluated by both outlier reduction and effective use of available low-bit levels. A natural next step is to extend this analysis beyond bell-shaped activation assumptions and develop transformations for broader distributional regimes. The findings of this paper should be interpreted within the scope of the evaluated settings. The method is motivated by activation patterns that have been widely observed in prior studies and are also present in the models evaluated in this paper. The practical effectiveness of INFOQUANT may vary when calibration conditions differ substantially from those used in evaluation.

Ablation Insights

The component ablation study shows that replacing a fixed Hadamard rotation with the learned PSOT rotation yields the largest single change in perplexity, suggesting that the core gain comes from learning where activation energy should be redistributed. The ordering of gains is important: first reshape the distribution, then reconstruct weights, then refine which calibration tokens and clipping ranges deserve attention. The weighting factor γ controls how strongly ASOT emphasizes outlier tokens in Eq. (11), with a moderate emphasis achieving the lowest perplexity. The block sizes of block-diagonal orthogonal matrices show that increasing the number of blocks results in a decline in quantization performance.

Additional Results

Results for Qwen-2.5 models show that INFOQUANT maintains superior performance on both the 14B and 32B models under 4-4-4 quantization. The full INFOQUANT results are reported in Table 11.

Limitations

The findings of this paper should be interpreted within the scope of the evaluated settings <ref:2605.

Improvements for AI systems

Peak Suppression Orthogonal Transformation (PSOT) reshapes activations into distributions that jointly have a smaller numerical range and greater dispersion within that range, which reduces quantization error by controlling the clipping scale. This allows low-bit quantizers to better match the underlying activation distribution.
Adaptive Outlier-Token Selection (ASOT) improves optimization robustness by selecting reliable outlier tokens based on sample-dependent inconsistencies, ensuring that the learned rotation generalizes across inputs rather than overfitting to fixed sequence positions.
Learnable Activation Clipping (LAC) refines the final quantization interval using parameters alpha and beta optimized by matching the quantized block output to the full-precision block output, leading to a more precise final quantization step size s(alpha, beta).
The improved system can be deployed with significant memory savings, such as achieving 3.25x memory saving for LLaMA-2 70B models under W4A4KV4 quantization compared to FP16.
The system maintains high performance on diverse tasks, preserving 97% of floating-point accuracy on average for LLaMA-2 and improving accuracy over rotation-based baselines in the W4A4KV4 setting.

Abstract

Low-bit activation quantization remains a major bottleneck in efficient large language model (LLM) deployment. The difficulty is not only that activations contain outliers, but that their distributions are often poorly matched to a low-bit uniform quantizer. Existing post-training quantization (PTQ) methods suppress peaks, balance channels, or minimize reconstruction error, yet they rarely specify what activation distribution is actually easy to discretize. As a result, activations may appear numerically smoother while still incurring large quantization error because the quantization range remains wide or most values collapse into a few levels near the mean. We recast activation transformation as quantizer-facing distribution design and analyze quantization error from an information-theoretic perspective. Our analysis shows that quantization-friendly activations should jointly have a smaller numerical range and sufficient dispersion within that range. Guided by this analysis, we propose InfoQuant, a train-free method that employs Peak Suppression Orthogonal Transformation (PSOT) to shape activations into more quantization-friendly distributions. We further introduce adaptive outlier-token selection to improve the robustness of PSOT during optimization. Across multiple LLM families, InfoQuant consistently outperforms prior PTQ and end-to-end training baselines. Under W4A4KV4, it preserves 97% of floating-point accuracy on average and reduces the LLaMA-2 13B performance gap by 42% over the previous state of the art. Code is available at [https://github.com/LLIKKE/InfoQuant](https://github.com/LLIKKE/InfoQuant)