A Comprehensive Evaluation on Quantization Techniques for Large Language Models

arXiv:2507.17417 · cs.LG · Submitted 2025-07-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "A Comprehensive Evaluation on Quantization Techniques for Large Language Models".

Jane: Here's the gist of the paper. The gist: This paper conducts an up-to-date and extensive review of state-of-the-art methods for post-training quantization in large language models, decoupling techniques into pre-quantization transformation and quantization error mitigation to ensure fair comparisons.

Tom: Decoupling Quantization Methods The authors first decouple the published quantization methods into two steps: pre-quantization transformation and quantization error mitigation The pre-quantization transformation step refers to preprocessing the data before quantization in order to mitigate the effect of outliers, making the data distribution smoother and more quantization-friendly The quantization error mitigation step involves applying techniques to compensate for the errors introduced by quantization, thereby reducing the overall quantization error and improving performance.

Jane: Pre-quantization Transformation Techniques The paper evaluates various pre-quantization transformations, focusing on how they reduce outliers in activations and weights The key pre-quantization transformations discussed include: Shifting: Shifting is a technique where the activation distribution is often asymmetric, and aligning the center of each channel can improve performance by quantizing activations per-token rather than along the channel dimension Outlier Suppression+ proposes shifting activation values to eliminate the asymmetry feature across channels.

Tom: Scaling: Scaling is a commonly used pre-quantization transformation method which scales activations in different input channels before quantization, and inversely scales the corresponding weight channels to make the results stay unchanged SmoothQuant first proposed using scaling to smooth the quantization difficulty from activations to weights.

Paper summary: Tom: So to wrap up, this paper, "A Comprehensive Evaluation on Quantization Techniques for Large Language Models," gives us a systematic experimental look at post-training quantization.

Jane: It’s really about showing that you can't just pick one method without looking at the whole picture because each solution has different components like pre-quantization transformation and error mitigation >

Lu: They’re emphasizing the importance of analyzing these connections to gain a deeper understanding of how all these existing methods work together >

Meng: So for me, the implication is that we can't just rely on one specific quantization scheme and need to understand how those different pieces fit together before we deploy anything at scale >

Jane: That’s right, and the authors are showing that they have done a fair and extensive investigation to ensure their comparisons are really sound on this ground >

Tom: The title of the paper is "A Comprehensive Evaluation on Quantization Techniques for Large Language Models," and it’s by Yutong Liu, Cairong Zhao, and Guosheng Hu >

Jane: In simple terms, they are telling us that to get good performance with model quantization you need to look at the whole process of transforming the data and then compensating for the errors >

Lu: They are emphasizing that this investigation remains critically important yet significantly underexplored in the research community because it’s a big area >

Meng: The implication is that we can't just rely on one specific quantization scheme and need to understand how those different pieces fit together before we deploy anything at scale >

Lalam: So for me, it means the method they recommend is a combination of optimized rotation and scaling with GPTQ and low-rank compensation >

Tom: So that’s the main configuration they found for achieving good results in this evaluation >

Jane: It boils down to understanding that performance depends on where you are in the process of quantizing, whether you’re looking at pre-quantization or error mitigation >

Conclusion: Tom: So we’ve been deep in these quantization techniques for large language models, and now we’re at the conclusion of this paper, "A Comprehensive Evaluation on Quantization Techniques for Large Language Models."

Jane: It really boils down to a thorough look at all the different ways you can shrink these massive models using post-training quantization.

Tom: Right, the authors are not just looking at one trick; they’ve done a comparison of pre-quantization transformations and error mitigation strategies.

Lu: They’re decoupling the process into two main steps, which is important because it shows how you handle outlier reduction separately from fixing the errors that creep in later.

Meng: From my side, I see that separating those steps is what makes this paper valuable for deployment because we have to manage both things at once when we move to production.

Lalam: The main finding is that combining rotation and scaling with low-rank compensation gave them the best performance on the Llama-three point two model they tested.

Jane: So what does that mean for us? It means that for a real system, you can’t just pick one method; you have to choose a combination of tools based on what the model and your hardware are doing.

Tom: Exactly. And they also looked at new formats like MXFP4 and NVFP4, finding that the format of the scaling factors has a big impact on performance when you’re dealing with those newer types of precision.

Lu: It suggests that we need to keep testing these different data formats because even if one format is advertised as better, how it handles scaling factors in practice matters a lot.

Meng: If I’m looking at running this on my setup, the inference cost analysis showed that adding that low-rank compensation adds overhead, about three point one percent in FP16 for the Llama-three point two model they used. That's a measurable hit on speed and memory we have to account for.

Jane: It’s important to remember that they also showed that using asymmetric quantization on activations gives better performance gains than trying to optimize the weights themselves in some cases.

Tom: So the title is about making sure you understand all these different components—the transformations, the errors, and the new formats—before you commit to a specific path for your LLM’s post-training quantization.

Jane: It leaves us with a clear idea that this field isn't just about finding one magic number; it’s about building a whole strategy from data smoothing to error correction.

Tom: And if you want to know what the next steps in this research look like, we’ll be looking at how these specific scaling factor formats impact performance across different model sizes in our next segment.

Yutong Liu, Cairong Zhao, Guosheng Hu

Tongji University · University of Bristol

cs.LG

Submitted: 2025-07-23

Updated: 2026-10-21

Comments: Accepted to Pattern Recognition

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 80/100

The gist: The gist: This paper conducts an up-to-date and extensive review of state-of-the-art methods for post-training quantization in large language models, decoupling techniques into pre-quantization

Key concepts

Pre-quantization Transformation: This step involves preprocessing data before quantization to smooth out distributions and reduce outliers. Techniques like shifting, scaling, and rotation are used to make the data more suitable for low-bit representation, aiming to improve the accuracy of the final quantized model.
Quantization Error Mitigation: These techniques are applied after quantization to compensate for performance losses introduced by rounding numbers. This can be done through self-compensation (analyzing loss errors) or low-rank compensation (using SVD to reconstruct errors), helping to maintain high accuracy despite using fewer bits.
Asymmetric Quantization: This method allows the zero point of quantization to shift dynamically based on the data's minimum and maximum values. Experimentation showed that applying this specifically to activations provides a significant performance boost compared to applying it only to weights, leading to tighter fits for skewed distributions.

Terminology

Quantization — 8× in this paper · explained in 15 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
Papers — 1× in this paper · explained in 3 episodes
Post-Training Quantization (PTQ) — 1× in this paper · explained in 2 episodes

Summary

The gist: This paper conducts an up-to-date and extensive review of state-of-the-art methods for post-training quantization in large language models, decoupling techniques into pre-quantization transformation and quantization error mitigation to ensure fair comparisons.

Decoupling Quantization Methods

The authors first decouple the published quantization methods into two steps: pre-quantization transformation and quantization error mitigation The pre-quantization transformation step refers to preprocessing the data before quantization in order to mitigate the effect of outliers, making the data distribution smoother and more quantization-friendly The quantization error mitigation step involves applying techniques to compensate for the errors introduced by quantization, thereby reducing the overall quantization error and improving performance.

Pre-quantization Transformation Techniques

The paper evaluates various pre-quantization transformations, focusing on how they reduce outliers in activations and weights The key pre-quantization transformations discussed include:

** Shifting:**

Shifting is a technique where the activation distribution is often asymmetric, and aligning the center of each channel can improve performance by quantizing activations per-token rather than along the channel dimension Outlier Suppression+ proposes shifting activation values to eliminate the asymmetry feature across channels.

** Scaling:**

Scaling is a commonly used pre-quantization transformation method which scales activations in different input channels before quantization, and inversely scales the corresponding weight channels to make the results stay unchanged SmoothQuant first proposed using scaling to smooth the quantization difficulty from activations to weights.

** Rotation:**

Rotation is a technique that adjusts data distribution by multiplying the data matrix by an orthogonal matrix, which largely changes the activation distributions, reduces outliers in the data, and thereby enhances quantization accuracy QuIP introduced rotation with random orthogonal matrices to reduce incoherence in weight-only quantization.

Quantization Error Mitigation Strategies

The paper analyzes the techniques used to offset the errors introduced during quantization, which are categorized into self-compensation and low-rank compensation

** Self-compensation:**

Self-compensation involves directly compensating for errors on the original weights by analyzing the loss error introduced by quantization and using its Hessian-based quadratic approximation Methods like OBQ [20] and GPTQ [8] compute the Hessian matrix used in Eq. (4) to find an optimal weight perturbation δ that minimizes the quantization loss error.

** Low-rank compensation:**

Low-rank compensation introduces an auxiliary branch dedicated to quantization error mitigation by introducing a low-rank term to reconstruct quantization error Methods such as ZeroQuant-v2 [17] and LQER [18] apply SVD to reconstruct the quantization error matrix E = W − dq(Wq).

Analysis of Quantization Settings

The evaluation systematically examines different settings that influence performance, including symmetry and granularity

** Symmetric vs. Asymmetric:**

Symmetric quantization forces the zero point to align with the floating point, resulting in a quantization range that is symmetric with respect to zero and enabling a very simple dequantization process. Asymmetric quantization allows the zero point to shift dynamically according to the minimum and maximum values of the data, which often leads to tighter fits to skewed distributions at the cost of slightly more complex computation. The results demonstrate that asymmetric quantization for activations leads to a significant improvement over symmetric quantization, while performance gain for weights is relatively minor.

** Granularity:**

The granularity of quantization often largely affects the low-bit quantization performance, with commonly adopted granularities including per-tensor, per-channel, or per-token Finer granularity typically leads to improved performance but also incurs higher memory overhead due to the need to store more quantization parameters.

FP4 Quantization and Variants

The study explores the potential of newer formats like MXFP4 and NVFP4, which are supported by NVIDIA’s GeForce RTX 50 Series architecture Both MXFP4 and NVFP4 store each value in an E2M1 format, and FP4 can better handle long-tail distributions compared to INT4. The paper finds that the format and precision of scaling factors have a significant impact on performance, noting that NVFP4 performs better than MXFP4 due to its smaller group size and more precise scaling factor format.

Conclusion on Optimal Configuration

The best performance is achieved when both optimized scaling and rotation are applied in conjunction with GPTQ and the low-rank compensation The optimal configuration yields a perplexity of 11.73 in per-channel quantization for the Llama-3.2-1B model, which is close to the BF16 baseline of 9.76. Furthermore, using asymmetric quantization on activations shows more benefits in performance compared to weights. The paper concludes that the format of scaling factors demonstrates a large effect on FP4 quantization.

Inference Cost Analysis

The inference cost analysis shows that the low-rank compensation introduces additional memory cost and computation, with an overhead computation ratio of 3.1% in FP16 for Llama-3.2-1B. The combination of optimized rotation and scaling with GPTQ and low-rank compensation demonstrates the best performance.

Evaluation Scope

The experiments are conducted on models such as LLaMA-3.2 (1B, 3B), LLaMA-3.1 (8B, 70B), and Qwen2.5-3B, evaluating their perplexity on the WikiText-2 dataset and assessing accuracy on several zero-shot evaluation tasks. The datasets utilized for evaluation include MMLU [33], SST2 [34], MathQA [35], ARC-C [36], HellaSwag [37], WinoGrande [38], BoolQ [39] and PIQA [41].

Implementation Details

For calibration, the GPTQ quantization and the low-rank compensation branch are calibrated using 128 samples from WikiText-2 with a sequence length of 2048 The training of the learnable orthogonal and scaling matrices follows the OSTQuant [4] procedure, using 1,000 samples from WikiText2 and optimized with RiemannAdam. For low-rank compensation, the rank k=32 is set and stored in FP16.

Data Format Comparison

The comparison of INT4, MXFP4, and NVFP4 shows that FP4 performs better than INT4 without prequantization transformation in per-channel setting For finer granularity scenarios, FP4 and INT4 show very small differences. NVFP4 performs better than MXFP4, showing a lower risk of noticeable accuracy drop.

Final Findings on FP8 Scaling

For FP8 format scaling factor, FP8 (E4M3) with an additional per-tensor FP32 scaling factor is better than FP8 (E8M0). The paper concludes that for small group sizes, using a 4-bit scaling factor for weights is also feasible, albeit with some loss in accuracy.

Overall Conclusion

The paper presents an experimental evaluation of large language model quantization by systematically investigating the effects of Post-Training Quantization (PTQ) on LLMs.

Improvements for AI systems

Optimized rotation and scaling yields best performance by applying both to pre-quantization transformation, which can be used on LLaMA-3.2-1B to achieve a perplexity of 11.73, close to the BF16 baseline of 9.76 on WikiText2 and achieves better zero-shot accuracy across various tasks.
Asymmetric quantization for activations is recommended because activation distributions tend to be asymmetric, leading to much better performance than fully symmetric quantization and reducing computational overhead during dequantization compared to fully asymmetric quantization for both weights and activations.
Using the combination of optimized rotation, scaling, GPTQ, and low-rank compensation achieves a perplexity of 11.73 in per-channel quantization for Llama-3.2-1B, which is close to the BF16 baseline of 9.76, demonstrating superior accuracy compared to GPTQ alone (perplexity of 11.8).
For FP4 quantization, using NVFP4 with E4M3 format and a two-level scaling scheme is better than MXFP4 because it performs better than MXFP4, showing a lower risk of noticeable accuracy drop, despite rotation-based methods yielding smaller improvements for MXFP4 and NVFP4 compared to INT4.
The system can utilize finer quantization granularity, such as per-group quantization with group sizes like 128, which yield superior model performance (lower perplexity) on WikiText2 at the expense of manageable storage overhead from scaling factors.

Abstract

For large language models (LLMs), post-training quantization (PTQ) significantly reduces memory and computational overhead. Despite rapid advancements, existing papers often lack experiments on the same ground. In addition, since quantization solutions typically contain multiple components, it is important to analyze the connections among numerous existing methods to gain an in-depth understanding of this field. To bridge these gaps, we conduct an up-to-date and extensive review of the state-of-the-art methods and perform comprehensive evaluations of low-bit quantization on representative text-only LLMs on the same ground to ensure fair comparisons. To our knowledge, this fair and extensive investigation remains critically important yet significantly underexplored in the research community. To better understand the connections, we first decouple published quantization into two steps: pre-quantization transformation to reduce outliers, and quantization error mitigation to offset performance loss. Second, we extensively evaluate the impact of quantization settings, including granularity and symmetry. Third, we analyze the performance of the latest MXFP4 and NVFP4 data formats. Our experimental results first demonstrate that optimizing rotation and scaling yields the best pre-quantization transformation, while combining low-rank compensation with GPTQ occasionally outperforms using GPTQ alone. Second, we verify that finer granularity improves performance at the cost of storage and asymmetric quantization benefits activations significantly more than weights. Third, we find that the format and precision of scaling factors have a significant impact on performance; notably, rotation-based transformation is effective for INT4 but demonstrates little improvement for MXFP4 and NVFP4, inspiring further investigation in this field.