Abstract Compression of Conversational Audio Context for LLM-based ASR

summary

The gist

The gist The authors show that conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities.

In short

Researchers investigated how conversational context from previous turns improves Large Language Model (LLM) speech recognition (ASR). They found that multi-turn training helps, especially for recognizing contextual entities. To make using raw context cheaper, they introduced Abstract Compression, which replaces long audio sequences with fixed latent tokens while keeping transcripts. This method recovers significant performance gains with a much smaller context footprint.

Key concepts

LLM-based ASR: This is the process where a large language model (like GPT) listens to spoken words and converts them into text. Standard systems treat each sentence in isolation, ignoring what was said before, which limits their ability to understand the full conversation.
Abstract Compression: A technique designed to reduce the cost of using long conversational history. Instead of feeding the entire audio from previous turns into the model, this method learns to condense that audio into a small set of fixed-size 'latent tokens' while explicitly saving and retaining the corresponding text transcript.
Contextual Entities: These are specific words or phrases in a conversation that only make sense when you know what was said earlier. For example, if someone says 'the book' in turn two, the context from turn one might tell the model exactly which book is being referred to. The paper shows compression helps these specific types of contextual clues.
Latent Tokens: These are a compact, learned representation of audio data. Instead of using every single audio sample as a separate piece of information, the model learns to represent the essential meaning or features of an entire prior turn using only a few fixed-length numerical tokens. This allows for efficient context storage.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

Abstract Compression of Conversational Audio Context for LLM-based ASR · Read on arXiv

Shashi Kumar, Esaûel Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Baęras-Roux, ×Hasindri Watawana, ×Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke

Idiap Research Institute · Uniphore, U.S.A. · University of Zurich, Switzerland · Brno University of Technology, Czech Republic

Standard LLM-based speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Here we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio embedding sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, a learned latent abstraction of prior-turn audio that replaces each prior-turn audio sequence with a fixed number of latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compression-enabled model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its tradeoffs.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Abstract Compression of Conversational Audio Context for LLM-based ASR".

Jane: Here's the gist of the paper. The gist The authors show that conversational multimodal context can improve LLM-based ASR, but only after supervised multi-turn training, with the largest gains appearing on contextual entities.

Jane: Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context Page 1>.

Paper summary: Tom: So, wrapping up "Abstract Compression of Conversational Audio Context for LLM-based ASR", the authors have shown that you can improve LLM based speech recognition by using conversational audio context, but only if you've supervised it on multi-turn data.

Jane: And the main contribution they make is proposing Abstract Compression, which lets us represent that history efficiently by compressing prior turn audio into a fixed number of latent tokens while explicitly keeping the transcripts.

Lu: The implications are that this offers a practical way to get part of the benefit from full raw context conditioning while significantly reducing the cost associated with processing long sequences in inference time.

Meng: For deployment, it means we can get better contextual accuracy than single-turn models without needing an astronomical amount of compute just to feed the conversation history into the model at once.

Lalam: Overall, their results suggest that compact representations of conversational audio context provide a quality efficiency trade-off for context aware ASR.

Tom: That's it then. It’s about finding that middle ground between having all the data and being able to run the system affordably.

Conclusion: Tom: So, this paper is about Abstract Compression of Conversational Audio Context for LLM-based ASR.

Jane: It’s about taking all that audio history from a conversation and shrinking it down into something smaller so an LLM can actually use it better.

Lu: The authors found that after some supervised multi-turn training, this context helps the AI recognize specific things in the conversation, like names or places.

Meng: But conditioning on the raw context is too much work because that audio sequence just gets really long very fast with a long chat.

Tom: That’s exactly what they tackle with Abstract Compression, which replaces those long audio sequences with a fixed number of learned latent tokens.

Jane: So, it keeps the transcripts but throws away most of the raw audio and replaces it with these compact tokens instead.

Lu: The mechanism uses a learnable cross-attention thing where turn-specific queries look at the original audio to create those compressed representations.

Meng: From what I see, they split their training into two stages, one just to align that compressed audio input correctly with the LLM space before doing the fine-tuning.

Tom: That makes sense, because you don't want a weird input format messing up the whole thing before you even start learning context.

Jane: And when they tested it on actual datasets, like DefinedAI and WoW, they saw that this compression method gets part of the benefit from using full context but with a much smaller audio footprint.

Lu: On the in-domain stuff, they saw Bias-WER drop from thirteen point five to thirteen point one with ten turns of context.

Meng: The WoW set was even more interesting, where they saw Bias-WER move down from twenty-five point six all the way to twenty three point three using this method.

Tom: So, the main conclusion is that you can get contextual improvements without having to process every single audio millisecond of every previous turn.

Jane: It’s a practical way to make context-aware ASR more affordable and efficient for real-world use cases.

More episodes

← Home