TTS-Q: Speaker-Aware Post-Training Quantization of Text-to-Speech Diffusion Transformers

arXiv:2610.12474 · cs.SD, cs.LG, eess.AS · Submitted 2026-08-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers".

Jane: The gist TTS-Q proposes a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation to improve speech quality in zero-shot text-to-speech diffusion transformers.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're talking about this paper now called "TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers." It sounds a bit technical, but the core idea is that when you compress these big text to speech models—these diffusion transformers—you have to make sure you don't just lose the words, but also not lose who is speaking.

Jane: Exactly. The authors are Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang and Chuanguang Yang. What they're tackling here is the problem that post-training quantization has to solve: you need to keep both the clarity of the speech and the specific voice identity.

Lu: The paper sets up this challenge by asking two main questions for calibration: which reference voices should we use, and how do we actually preserve that voice identity when we quantize the model? It's about picking which voices matter most and then figuring out a way to lock in the speaker's characteristics during the compression process.

Meng: If I look at this from a deployment side, what I need to know is how much extra effort this adds during that post-training phase. Does it make things significantly slower than just doing standard quantization?

Lalam: The gist of what they propose is TTS-Q, which is a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation >

The paper's summary: Tom: Okay, so the core mechanism involves two main parts. First, they use coverage-guided calibration sampling to choose the right conditions for reconstruction. They look at the speaker embedding space and select reference conditions that give good coverage under a fixed calibration budget F >

Jane: So, how do those conditions relate to quality? The paper says they measure distance in this speaker embedding space using something like farthest-point or average-distance cost under a fixed budget K >

Lu: They define the distance r of a sample S to the nearest selected condition as r(S) equals the minimum d of i in P, where d is max(zero one minus u i transpose u j) >

Meng: That sounds like they are trying to find voices that are well-represented in the embedding space before they start quantizing. What's the actual benefit there? Is it just better fidelity on average?

Lalam: They use two geometric objectives for cost minimization: Cmax(S), which emphasizes the least covered reference embedding, and Cavg(S), which gives more weight to densely populated regions >

The paper's improvements: Tom: Then they move to the second stage, speaker identity distillation. This is where they use the full-precision teacher's synthesized speech as an identity target for the student model. They minimize a cosine dissimilarity loss Lid(c; ϕ) = D E(Geϕ(c), sg

E(G(c)): ) >

Jane: That loss function is key because it’s not just comparing the final audio waveforms directly. It compares the speaker embeddings generated by both the teacher and student after they've gone through their respective generators >

Lu: This means they are trying to preserve voice identity without needing perfect sample-wise waveform agreement, which is a big step forward since we don't always have access to those perfect teacher states >

Meng: So, this is about making sure the student model learns the *style* of the voice from the teacher's full precision output rather than just matching every single tiny detail in every reconstruction unit >

Lalam: They combine these two into one final calibration objective: Lcal(c; ϕ) = Lrec(c; ϕ) + lambda Lid(c; ϕ), where lambda is a non-negative weight that controls how much they care about the identity term over the reconstruction loss >

Conclusion: Tom: So, to wrap up, TTS-Q uses coverage-guided calibration sampling to pick good reference conditions in speaker embedding space and then adds speaker identity distillation to supervise the quantization parameters using a weighted loss function >

Jane: The practical outcome they show is that this method consistently achieves higher speaker similarity SIM and predicted naturalness UTMOS than external baselines on W4A4 >

Lu: They found that the selection strategy matters, for instance, farthest-point selection performed better than average-distance selection on all reported metrics for LongCat-1B > <ref:2610.12474#pg1>

Meng: From a deployment standpoint, they also show some speed improvements. For LongCat models, it runs one point six six to two point three four times faster during warm inference when using INT8 execution on H100s >

Lalam: And finally, they noted that the choice of calibration objective changes depending on the model; farthest-point selection was chosen for LongCat-1B while average-distance selection was used for LongCat-3 point 5B and F5-TTS based on development set speech quality >

Wyze Inc. · City College of New York, City University of New York · Institute of Computing Technology, Chinese Academy of Sciences · National Taiwan University · Tsinghua University

cs.SD, cs.LG, eess.AS

Submitted: 2026-08-11

Updated: 2026-10-20

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist TTS-Q proposes a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation to improve speech quality in zero-shot

Key concepts

Post-Training Quantization (PTQ): This is a technique used to shrink large AI models by reducing the precision of their numerical values after they have been fully trained. While it saves memory and speeds up inference, standard PTQ often causes quality loss in speech models because it doesn't account for how different numbers affect human hearing.
Coverage-Guided Calibration Sampling: This is a method for selecting the best audio samples to calibrate the model during quantization. Instead of picking random samples, this technique uses speaker embeddings to ensure that the selected conditions cover a wide range of speaker identities, minimizing errors across all voices within a set budget.
Speaker Identity Distillation: This is a secondary calibration step designed specifically to preserve the unique voice characteristics after quantization. It forces the quantized model's output to match the style and timbre of high-quality, full-precision speech generated by a teacher model, ensuring speaker identity is maintained.
W4A4 PTQ Study: This refers to a specific experimental setup where the 1B and 3.5B diffusion transformer models are quantized using only 4 bits for both weights and activations. The study investigates how this extreme compression affects speech quality, showing that TTS-Q successfully mitigates the resulting loss of speaker timbre.

Terminology

Quantization — 1× in this paper · explained in 15 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist TTS-Q proposes a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation to improve speech quality in zero-shot text-to-speech diffusion transformers.

How it works

Coverage selection is guided by the speaker embedding space, where the distance to selected conditions is measured using a metric like farthest-point or average-distance cost under a fixed budget K.
Initial calibration involves local reconstruction, minimizing lb(ϕ; hb) = ∥fq b,ϕ(hb) − fb(hb)∥2omega max(∥fb(hb)∥2omega, ϵ), which uses cached teacher inputs for every reconstruction unit.
Speaker identity distillation is performed in a second calibration stage using the full-precision teacher's synthesized speech as an identity target, minimizing the cosine dissimilarity loss Lid(c; ϕ) = D E(Geϕ(c),sg[E(G(c))]).
The final calibration objective combines these terms: Lcal(c; ϕ) = Lrec(c; ϕ) + λLid(c; ϕ), where λ ≥ 0 weights the identity term, and only quantization parameters are updated.

Coverage-Guided Calibration Sampling

The method selects reference conditions by their coverage in the speaker embedding space under a fixed calibration budget K. The distance ri(S) to the nearest selected condition is defined as ri(S) = min j∈S d ij, where d ij = max(0, 1 − u i⊤u j). Two geometric objectives are used for cost minimization: Cmax(S) = max i∈P ri(S), emphasizing the least covered reference embedding, and Cavg(S) = 1/P Σ i∈P ri(S), which gives more weight to densely populated regions. The paper reports that farthest-point sampling adds the unselected candidate with the largest r j(S) for Cmax, while average-distance selection starts at the medoid and adds the candidate that minimizes Cavg(S U j).

Speaker Identity Distillation

Speaker identity distillation uses speech generated by the full-precision teacher as an identity target to preserve voice after quantization. The identity loss is defined as Lid(c; ϕ) = D E(Geϕ(c),sg[E(G(c))]), where D is the cosine dissimilarity, which directs the loss toward voice identity without requiring sample-wise waveform agreement. This stage starts from the checkpoint obtained by local reconstruction and updates parameters with Lcal(c; ϕ) = Lrec(c; ϕ) + λLid(c; ϕ), where λ ≥ 0 weights the identity term. Identity gradients pass through the frozen speaker encoder, waveform decoder, and final two sampling updates to optimize quantization parameters using a straight-through estimator.

Experimental Results and Comparisons

TTS-Q consistently achieves higher speaker similarity (SIM) and predicted naturalness (UTMOS) than evaluated external baselines at W4A4. For instance, on LongCat-3.5B, TTS-Q achieved a UTMOS of 3.457 compared to 2.801 for the full precision model. The selection strategy significantly impacts quality; Farthest-point selection performed better than average-distance selection on all reported metrics for LongCat-1B. Furthermore, adding identity supervision raises SIM point estimates across all settings relative to reconstruction continuation with reconstruction alone.

Deployment and Storage Evaluation

In deployment tests, LongCat runs 1.66–2.34× faster than its native FP32 reference during warm inference. F5 remains slower than its native FP16 reference, but the paper notes that F5 fused and original W4A4 paths produce sample-identical audio on all 60 conditions. INT4 packing reduces complete-pipeline storage by 1.74–3.10× relative to the FP16 weight-storage reference.

Limitations

The paper notes several limitations, including the need for listening tests to confirm automatic metrics, generalization remains untested across other pools and budgets, and identity gradients only cover the final two sampling updates. Furthermore, the effects of approximations like teacher-state reuse in F5 on final quality have not been isolated. The deployment measurements cover warm H100 inference on 60 conditions, and full-test quality with integer execution remains unverified.

The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4. All methods use W4A4 for the same DiT linear layers, with other operations in native precision. The choice of coverage objective depends on the model, with farthest-point selection chosen for LongCat-1B and average-distance selection for LongCat-3.5B and F5-TTS based on development set speech quality. The paper concludes that TTS-Q combines calibration conditions selected for speaker coverage with identity supervision from a full-precision teacher to achieve higher speaker similarity and predicted naturalness than evaluated external baselines at W4A4.

Improvements for AI systems

Speaker-Aware Post-Training Quantization (TTS-Q) for Zero-Shot TTS Diffusion Transformers (DiTs): Improved Speaker Identity Preservation in Compressed Models. This method combines coverage-guided calibration sampling with speaker identity distillation to ensure that post-training quantization must preserve both intelligibility and speaker identity. The improved system can generate speech from a short reference recording while maintaining high speaker similarity and predicted naturalness, consistently outperforming random selection methods.
Coverage-Guided Calibration Sampling for Optimal Quantization Condition Selection. By selecting reference conditions based on their coverage in speaker embedding space, the system avoids suboptimal compression artifacts. This improves quality by choosing conditions that better represent the candidate voices, as demonstrated by farthest-point sampling or average-distance strategies, which show improvements over random selection in both English and Mandarin metrics.
Speaker Identity Distillation for Fine-Grained Quantization Supervision. The system uses the full-precision teacher’s synthesized speech to supervise voice identity, quantified by the loss function Lid(c; ϕ) = D[E(Geϕ(c)), sg[E(G(c))]]. This allows the quantization parameters to be updated based on the reference recording's voice, which is crucial for preserving identity beyond simple local reconstruction.
Deployment Acceleration and Efficiency for LongCat Models. The system enables warm inference acceleration of 1.66–2.34× over native FP32 reference during deployment tests using INT8 execution on H100s with CUDA Graphs, significantly reducing latency and memory allocation while maintaining quality metrics like SIM and UTMOS.
Model-Specific Calibration Strategy Selection for Enhanced Accuracy. The system dynamically chooses the calibration objective based on the model: farthest-point selection performs better than average-distance selection on all reported metrics for LongCat-1B and average-distance selection performs better on all metrics for F5-TTS. This tailored approach ensures that the calibration set maximizes quality gains specific to each architecture.

Abstract

Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.