WUSH: Near-Optimal Adaptive Transforms for LLM Quantization

The gist The construction WUSH combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably nearoptimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation.

Problem Setup and Goal

The paper addresses the difficulty that a few extreme “outlier” weights and activations expand the dynamic range and thereby degrade the effective resolution of low-bit representations. The central difficulty in model deployment is that a few extreme “outlier” weights and activations expand the dynamic range and thereby degrade the effective resolution of low-bit representations. The objective is to choose transforms TW and TX such that the L2 output loss l = d−1 outd−1 batch q (TWW)⊤ q (TXX) − W⊤X is minimized under the given quantizer q. This involves minimizing the blockwise losses l(i) independently, where l(i) is defined as d−1 outd−1 batch q TW(i)W(i)⊤ q TX(i)X(i) − W⊤ (i)X (4).

Optimal Transform Derivation

The paper derives closed-form optimal linear blockwise transforms for joint weight-activation quantization, which are generally non-orthogonal and adaptive. The construction WUSH is derived by showing that the optimal TW(i) and TX(i) can be constructed as Txvsh(i) = HS− 1/2 (i) V⊤ (i)X′(i), Twush(i) = HS− 1/2 U⊤ (i)W′(i), respectively. This construction "combines a Hadamard backbone with a datadependent second-moment component to form a non-orthogonal transform that is provably nearoptimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation". The Hadamard matrix H is the only data-agnostic ingredient in our optimal formulation in Eq. (7).

Theoretical Analysis of Optimality

The theoretical derivation shows that for floating-point (FP) data types, the optimal configuration of T for floating-point (FP) data types is U′ = H, S′ = S1/2, and R = I. For integer (INT) types, the same configuration is optimal up to a d o(1) factor for zeromean multivariate Gaussian/Laplacian distributed data and within a d factor for any distribution. The analysis shows that WUSH more effectively reduces the error by aligning the ellipse’s major axis to the error reduction (blue) regions. Specifically, in the FP case, WUSH can reduce the error by at most d times in the extreme outlier scenario.

Empirical Performance and Efficiency

Empirically, WUSH significantly improves end-to-end accuracy over prior transform-based baselines. For Llama-3.1-8BInstruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ. Despite using a distinct (data-aware) transform per block, WUSH admits an efficient fused GPU kernel whose throughput matches that of optimized blockwise Hadamard kernels while delivering up to 5.8× per-layer speedups over BF16 via FP4 MatMul. The online transform is fused with activation quantization in a kernel detailed in Section 5, leading to negligible runtime overhead.

Experimental Results

WUSH consistently yields the smallest loss for MXFP4 and INT4. For Llama-3.1-8BInstruct on W4A4 accuracy, GPTQ-WUSH improves by up to +0.7 average points. The results show that WUSH improves the state-of-the-art for weight-activation quantization. The performance comparison in Figure 2 shows that WUSH + Quant + FP4 MatMul achieves speedups of up to 5.8× over BF16, and it closely matches the performance of the optimized H + Quant + FP4 MatMul baseline.

Conclusion

WUSH is a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU kernel implementations. This work clarifies the empirical success of Hadamard rotations and indicates that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The results conclude that WUSH improves the state-of-the-art for weight-activation quantization. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU kernel implementations. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation. The results clarify the empirical success of Hadamard rotations and indicate that principled, data-aware block transforms can yield significant improvements, e.g., making the MXFP format competitive with NVFP. The construction WUSH combines a fixed Hadamard backbone with a data-aware component derived from second-order statistics, yielding a non-orthogonal blockwise transform that is provably nearoptimal and remains amenable to efficient GPU implementation.

1. Data-aware blockwise transforms for weight-activation quantization can be derived using WUSH, which combines a Hadamard backbone with a datadependent second-moment component to form a non-orthogonal transform that is provably nearoptimal for FP and INT quantizers. This allows the system to achieve accuracy improvements of up to +2.8 average points over strong Hadamard-based baselines on Llama-3.1-8B-Instruct in MXFP4 RTN.

2. The WUSH construction enables a significant throughput increase, delivering up to 5.8x per-layer speedups over BF16 via FP4 MatMul. This means the system can process LLM inferences much faster on compatible hardware while maintaining high accuracy.

3. By utilizing the WUSH transform in conjunction with GPTQ, the system gains improved accuracy, specifically achieving +0.7 points on Llama-3.1-8B-Instruct in MXFP4 GPTQ comparisons. This makes the system more robust when using iterative weight update methods during quantization.

4. The proposed method allows for a fused GPU kernel implementation where the online transform is integrated with activation quantization, resulting in negligible runtime overhead. This ensures that the performance gains are realized without incurring significant latency during inference.

5. The system can be optimized for both FP and INT formats using WUSH, providing closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floatingpoint formats. This versatility allows the model to adapt its transformation strategy based on the target precision.

Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization errors. Prior transform-based mitigations (e.g., Hadamard rotations) are fixed and data-agnostic, and their optimality for quantization has remained unclear. We derive closed-form optimal linear blockwise transforms for joint weight-activation quantization under standard RTN AbsMax-scaled block quantizers, covering both integer and floating-point formats. The resulting construction, WUSH, combines a Hadamard backbone with a data-dependent second-moment component to form a non-orthogonal transform that is provably near-optimal for FP and INT quantizers under mild assumptions while admitting an efficient fused GPU implementation. Empirically, WUSH improves W4A4 accuracy over the strongest Hadamard-based baselines (e.g., on Llama-3.1-8B-Instruct in MXFP4, it gains +2.8 average points with RTN and +0.7 with GPTQ) while delivering up to 5.8 times per-layer throughput over BF16 via FP4 MatMul. Source code is available at https://github.com/IST-DASLab/WUSH.