Abstract Compression of Conversational Audio Context for LLM-based ASR

arXiv:2603.26246 · cs.CL, cs.AI, cs.LG, eess.AS · Submitted 2026-03-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Abstract Compression of Conversational Audio Context for LLM-based ASR".

Jane: Here's the gist of the paper. The gist The authors show that conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities.

Jane: Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context Page 1>.

Paper summary: Tom: So, wrapping up "Abstract Compression of Conversational Audio Context for LLM-based ASR", the authors have shown that you can improve LLM based speech recognition by using conversational audio context, but only if you've supervised it on multi-turn data.

Jane: And the main contribution they make is proposing Abstract Compression, which lets us represent that history efficiently by compressing prior turn audio into a fixed number of latent tokens while explicitly keeping the transcripts.

Lu: The implications are that this offers a practical way to get part of the benefit from full raw context conditioning while significantly reducing the cost associated with processing long sequences in inference time.

Meng: For deployment, it means we can get better contextual accuracy than single-turn models without needing an astronomical amount of compute just to feed the conversation history into the model at once.

Lalam: Overall, their results suggest that compact representations of conversational audio context provide a quality efficiency trade-off for context aware ASR.

Tom: That's it then. It’s about finding that middle ground between having all the data and being able to run the system affordably.

Conclusion: Tom: So, this paper is about Abstract Compression of Conversational Audio Context for LLM-based ASR.

Jane: It’s about taking all that audio history from a conversation and shrinking it down into something smaller so an LLM can actually use it better.

Lu: The authors found that after some supervised multi-turn training, this context helps the AI recognize specific things in the conversation, like names or places.

Meng: But conditioning on the raw context is too much work because that audio sequence just gets really long very fast with a long chat.

Tom: That’s exactly what they tackle with Abstract Compression, which replaces those long audio sequences with a fixed number of learned latent tokens.

Jane: So, it keeps the transcripts but throws away most of the raw audio and replaces it with these compact tokens instead.

Lu: The mechanism uses a learnable cross-attention thing where turn-specific queries look at the original audio to create those compressed representations.

Meng: From what I see, they split their training into two stages, one just to align that compressed audio input correctly with the LLM space before doing the fine-tuning.

Tom: That makes sense, because you don't want a weird input format messing up the whole thing before you even start learning context.

Jane: And when they tested it on actual datasets, like DefinedAI and WoW, they saw that this compression method gets part of the benefit from using full context but with a much smaller audio footprint.

Lu: On the in-domain stuff, they saw Bias-WER drop from thirteen point five to thirteen point one with ten turns of context.

Meng: The WoW set was even more interesting, where they saw Bias-WER move down from twenty-five point six all the way to twenty three point three using this method.

Tom: So, the main conclusion is that you can get contextual improvements without having to process every single audio millisecond of every previous turn.

Jane: It’s a practical way to make context-aware ASR more affordable and efficient for real-world use cases.

Shashi Kumar, Esaûel Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Baęras-Roux, ×Hasindri Watawana, ×Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke

Idiap Research Institute · Uniphore, U.S.A. · University of Zurich, Switzerland · Brno University of Technology, Czech Republic

cs.CL, cs.AI, cs.LG, eess.AS

Submitted: 2026-03-27

Updated: 2026-10-20

Comments: 7 pages; accepted to IEEE SLT 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The authors show that conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities.

Key concepts

LLM-based ASR: This is the process where a large language model (like GPT) listens to spoken words and converts them into text. Standard systems treat each sentence in isolation, ignoring what was said before, which limits their ability to understand the full conversation.
Abstract Compression: A technique designed to reduce the cost of using long conversational history. Instead of feeding the entire audio from previous turns into the model, this method learns to condense that audio into a small set of fixed-size 'latent tokens' while explicitly saving and retaining the corresponding text transcript.
Contextual Entities: These are specific words or phrases in a conversation that only make sense when you know what was said earlier. For example, if someone says 'the book' in turn two, the context from turn one might tell the model exactly which book is being referred to. The paper shows compression helps these specific types of contextual clues.
Latent Tokens: These are a compact, learned representation of audio data. Instead of using every single audio sample as a separate piece of information, the model learns to represent the essential meaning or features of an entire prior turn using only a few fixed-length numerical tokens. This allows for efficient context storage.

Summary

The gist The authors show that conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities.

How it works

Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context Page 1>.
After supervised fine-tuning on multi-turn inputs, the model does benefit from context, with the clearest gains appearing on contextual entities, as reflected in Bias-WER (defined in Section 6.2) Page 1>.
To address the cost of raw-context conditioning, Abstract Compression is introduced, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly Page 1>.

Abstract Compression for Context-Aware ASR

Abstract Compression compresses the audio from prior turns into a small set of latent tokens, while retaining prior-turn transcripts in their original textual form Page 2>.
This design targets the dominant source of context-token cost in LLM-based ASR, since audio is represented by long token sequences whereas transcript is comparatively compact Page 5>.
The compression mechanism involves learning a function g(·) that maps a variable-length audio sequence to a fixed number of latent tokens Page 5>.
This function is implemented as a learnable cross-attention mechanism with turn-index-specific query matrices Qi, which are defined for relative context positions with respect to the current turn Page 5>.
The compressed representation Zi is obtained by cross-attending Qi to the original audio tokens of that turn, i.e., Zi = CrossAttention(Qi, Xi, Xi) Page 5>.
Each context turn Ti in Eq. 3 is replaced by its compressed counterpart T'i: T'i = ⟨user⟩Zi⟨end⟩⟨assistant⟩Yi⟨end> Page 5>.
The resulting compressed multi-turn input sequence is Scomp = T'1T'2... T'N-1 ⟨user⟩XN ⟨end> Page 5>.

Training Strategy

A two-stage training strategy is adopted following Lin et al. (2025) Page 5>.
Stage 1, Compressed-Audio-to-LLM Alignment, repurposes the single-turn ASR task by replacing the raw audio tokens X1 in the baseline prompt (Eq. 2) with the compressed tokens Z1 Page 5>.
During Stage 1, the base model is frozen, and only the compression module, including turn-specific queries Qi and the cross-attention parameters, is optimized Page 5>.
Stage 2 performs contextual fine-tuning using the compressed multi-turn input Scomp in Eq. 7 Page 5>.
In Stage 1, the goal is to make the compressed audio representations compatible with the LLM input space by predicting transcript Y1 from compressed audio input Page 5>.
In Stage 2, the model jointly optimizes the compression module and the audio LoRA parameters of the LLM, allowing it to combine compressed context turns with the full-resolution audio of the current turn Page 5>.

Experimental Results

Raw multi-turn conditioning improved over single-turn baseline on both test sets after multi-turn fine-tuning, with clear gains on entity words Page 7>.
On the in-domain DefinedAI set, using 10 context turns reduces Bias-WER from 13.5% to 13.1%, while WER changes slightly from 7.6% to 7.5% Page 7>.
On the more entity-dense WoW set, the gains are larger: WER improves from 13.4% to 12.7% and Bias-WER from 25.6% to 23.3% Page 7>.
Abstract Compression improves Bias-WER relative to the single-turn baseline while remaining slightly behind raw-context conditioning on the in-domain DefinedAI set Page 7>.
On WoW, the compressed model reaches 24.5% in Bias-WER, compared with 25.6% for the single-turn baseline, and 23.3% for raw-context conditioning Page 7>.
The compression rate rho audio n measures how aggressively the audio is compressed, whereas rho context n captures the realized reduction in the full conversational context seen by the model at inference time Page 7>.

Ablation Studies

The number of latent tokens K directly controls the information capacity of the bottleneck, and larger K makes compressed context more useful in Stage 2 Page 8>.
With K = 16, Bias-WER drops from 26.5% without context to 24.5% with 5 or 10 context turns, suggesting that very aggressive compression removes information needed for cross-turn disambiguation Page 8>.
The Stage 1 results show that a larger Stage 1 training set clearly helps in the single-turn alignment setting, but this gain does not necessarily translate into better contextual ASR after Stage 2 fine-tuning Page 8>.
For K = 16 and DefinedAI Stage 1 training, Bias-WER drops from 26.5% with zero context turns to 25.3% with one turn and 24.5% with five turns, with no additional improvement with ten turns Page 8>.
This pattern suggests that a moderate amount of compressed context captures most of the available contextual benefit, as gains largely saturate by five turns Page 8>.

Conclusion

Conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities Page 8>.
Abstract Compression reduces the cost of raw-context conditioning by compressing prior-turn audio into a fixed latent budget and recovers part of the benefit of full raw context Page 8>.
Overall, our results suggest that compact representations of conversational audio context offer a practical quality-efficiency trade-off for context-aware ASR Page 8>.

Limitations

We evaluate Abstract Compression only in an audio-compression setting with a single multimodal LLM backbone Page 9>.
Our efficiency analysis focuses on input token budget and promptlength reduction, rather than direct measurements of inference latency, memory usage, or KV-cache behavior Page 9>.

References

Abdelrahman Abouelenin et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras arXiv preprint arXiv:2503.01743 Page 9>.
Jean-Baptiste Alayrac et al. 2022. Flamingo: a visual language model for few-shot learning In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA.
Andrés Carofilis et al. 2026. Text-only adaptation in llm-based asr through text denoising Preprint arXiv:2601.

Improvements for AI systems

Abstract Compression allows LLM-based ASR to condition on prior conversational context by replacing raw audio portions with a fixed number of learned latent tokens while explicitly retaining transcripts, thus reducing the high key-value (KV) cache cost, increased latency, and memory bottlenecks during inference.
The system can improve the recognition of contextual entities because after supervised multiturn training, conversational context mainly helps with the recognition of contextual entities.
The improved system maintains performance gains under a smaller budget by using Abstract Compression to recover part of the gains of raw-context conditioning with a smaller prior-turn audio footprint.
The system can be trained efficiently using a two-stage training strategy: Stage 1 aligns compressed audio representations to the LLM through single-turn ASR, and Stage 2 performs curriculum-based fine-tuning on compressed multi-turn inputs.
The system is optimized for efficiency by measuring the overall context compression rate rho context n, which captures the realized reduction in the full conversational context seen by the model at inference time, reflecting a smaller token budget for long contexts.

Abstract

Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Here we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio embedding sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, a learned latent abstraction of prior-turn audio that replaces each prior-turn audio sequence with a fixed number of latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compression-enabled model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its tradeoffs.