Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

arXiv:2607.05612 · cs.CL, eess.AS · Submitted 2026-07-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition".

Jane: Here's the gist of the paper. The gist The relation between language model (LM) perplexity and automatic speech recognition (ASR) word error rate for modern end-to-end ASR is revisited, challenging historical assumptions about a linear log-log relationship due to internal language modeling capacity in modern systems and the integration of large language models.

Tom: Problem Formulation The paper investigates the relation between LM perplexity and ASR WER by studying whether external LMs still improve modern end-to-end ASR over a no-LM baseline, if the PPL-WER relation remains approximately linear in loglog space, and how internal language modeling (ILM) subtraction affects the observed PPL-WER relation for AED systems.

Jane: The core relationship is modeled by log(WER) = α · log(PPL) + β, where α measures sensitivity to relative changes in PPL, and β captures the recognition setup-dependent offset ASR and LM Integration The paper examines how external LMs are integrated into different ASR architectures: 1.

Tom: CTC ASR systems use frame-synchronous decoding with a log-linear score over frame-level CTC labels and a sequence-level LM score, where the LM state is advanced only when non-blank labels are emitted 2.

Paper summary: Tom: So, we’re wrapping up this look at "Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition."

Jane: Basically, they’re saying that the old simple rules about how language model perplexity relates to word error rate don't hold up anymore.

Lu: They are testing if external language models still help these new end-to-end ASR systems, especially when you look at how internal language modeling capacity works inside the models.

Meng: What they found is that whether you have an external language model or not, the relationship between perplexity and error rate isn't one single log-log trend anymore; it depends on the setup.

Lalam: They specifically looked at CTC systems and AED systems to see how different things, like context length and internal language model subtraction, change that relationship.

Tom: So for a listener just driving, the main thing is that you can't just look at one formula anymore; you need to know what kind of system you're dealing with.

Jane: Exactly, because for attention-based systems, subtracting the internal language model helps show how much the external language model actually adds value to the final error rate.

Lu: It shows that when you have a lot of acoustic context length, the external LM gain is often smaller compared to when the acoustic context is short.

Meng: From a deployment standpoint, this means we can't just rely on one benchmark; we have to consider our specific model architecture and how much internal language modeling capacity it has built in.

Lalam: And for large language models, they found that word-level perplexity is tricky to interpret because of the different tokenization scales you’re dealing with.

Jane: It’s a complex picture, but the main point is that we need to be more nuanced about how these different components interact in modern ASR pipelines.

Tom: That complexity means we still have a lot of work ahead figuring out the best way to deploy these things reliably.

Conclusion: Tom: So, we're wrapping up our look at this paper, "Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition."

Jane: The authors are basically saying that we have to stop relying on one simple equation that used to work for language model perplexity and word error rate in speech recognition.

Lu: They are challenging the old linear log-log relationship because modern end-to-end systems already have internal language modeling power built in, and they test if external language models still add value.

Meng: What this means for us at scale is that you can't just plug in a benchmark number and assume it works across all different model setups anymore.

Lalam: They looked at everything from CTC systems to attention-based encoder decoder systems, and they showed that context length matters for how much the external language model helps.

Tom: So, the main thing is that external language models still help these systems, but you have to look at context length and whether you're using internal language model subtraction when you're measuring that help.

Jane: Right, so we can’t just take old assumptions about how perplexity relates to error rate and apply them blindly to these new end-to-end models without accounting for all those internal complexities.

Lu: That complexity is huge because it shows that the relationship isn't a single global trend; its behavior changes depending on the architecture you pick.

Tom: It’s a signal that we need more nuanced ways to measure performance when deploying these ASR systems in real life.

Jane: And this leads us into how those large language models fit into this picture, especially when you look at word-level perplexity versus subword PPL.

Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang, Robin Schmitt, Ralf Schluter, Hermann Ney

AppTek.ai GmbH · Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University

cs.CL, eess.AS

Submitted: 2026-07-06

Updated: 2026-10-18

Comments: Accepted at IEEE SLT 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist The relation between language model (LM) perplexity and automatic speech recognition (ASR) word error rate for modern end-to-end ASR is revisited, challenging historical assumptions about a

Key concepts

Language Model Perplexity (PPL): PPL measures how well a language model predicts a sequence of words; lower PPL indicates the model is more confident and accurate in its predictions. In ASR, it's used as an estimate for recognition quality, but modern systems complicate this due to internal language modeling.
End-to-End ASR Systems: These are modern speech recognition models that perform acoustic and language modeling simultaneously within a single neural network. Unlike older systems, these models have built-in language modeling capabilities, which changes how external language models affect performance.
Internal Language Modeling (ILM) Subtraction: This technique involves subtracting the perplexity contribution of the model's internal language model from the total score. This helps researchers isolate and measure the specific impact of an *external* language model on ASR performance, especially in attention-based decoder systems.
Log-Log Relation: This refers to a mathematical relationship where both variables are plotted on logarithmic scales. Historically, PPL and WER were thought to follow a linear trend in this space. The paper investigates whether this simple linear trend still holds true for modern ASR systems.

Terminology

Decoder — 2× in this paper · explained in 2 episodes
Perplexity — 2× in this paper · explained in 3 episodes
Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 9 episodes

Summary

The gist The relation between language model (LM) perplexity and automatic speech recognition (ASR) word error rate for modern end-to-end ASR is revisited, challenging historical assumptions about a linear log-log relationship due to internal language modeling capacity in modern systems and the integration of large language models.

Problem Formulation

The paper investigates the relation between LM perplexity and ASR WER by studying whether external LMs still improve modern end-to-end ASR over a no-LM baseline, if the PPL-WER relation remains approximately linear in loglog space, and how internal language modeling (ILM) subtraction affects the observed PPL-WER relation for AED systems. The core relationship is modeled by log(WER) = α · log(PPL) + β, where α measures sensitivity to relative changes in PPL, and β captures the recognition setup-dependent offset

ASR and LM Integration

The paper examines how external LMs are integrated into different ASR architectures:

CTC ASR systems use frame-synchronous decoding with a log-linear score over frame-level CTC labels and a sequence-level LM score, where the LM state is advanced only when non-blank labels are emitted
AED ASR models use a log-linear score that includes an internal language model (ILM) subtraction term: aˆSˆ1 = arg max aS+1 log pAED(aS+1xT1) + λ log pLM(aS+1) − γ log pILM(aS+1)

Experimental Setup

The study evaluates two ASR tasks: LibriSpeech, which has a large external LM corpus (about 80 times more than the acoustic transcriptions), and AppTek Spanish, which involves spontaneous speech with a smaller external LM text ratio The models tested include Conformer encoders for CTC and AED systems, and various LM families such as Transformer LMs, LSTM LMs, and n-gram LMs

Key Findings on PPL-WER Relation

The PPL-WER relation is not a single global loglog trend; it exhibits different behaviors depending on the setup:

- CTC systems show a clear reduction in slope with subword-level PPL, with the slope decreasing from α = 0.267 to α = 0.092 on dev-other for LibriSpeech and a similar decrease on test-other The relation remains approximately linear up to a PPL split point of 50, with high-PPL slopes remaining above zero.

- For AppTek Spanish CTC, the slope becomes smaller in the high-PPL region, fitting up to PPL 450 with α = 0.046 and above with α = 0.014 This is attributed to the ratio between external LM text and ASR transcription text being about 6 for AppTek Spanish compared to about 80 for LibriSpeech.

- The effect of encoder context length shows an interaction where the external LM gain is larger for shorter context lengths, suggesting the external LM becomes more important when the encoder has less acoustic context.

Effect of Internal Language Model Subtraction

For AED ASR models, applying ILM subtraction increases the slope from 0.19 to 0.29 in the low-perplexity regime, indicating that ILM subtraction strengthens the correlation between perplexity and WER and increases the gain from external LMs Furthermore, this suggests that saturation effects at high perplexities are likely due to the ILM dominating predictions when external LM perplexity is higher than the ILM perplexity

Large Language Model (LLM) Analysis

LLM experiments reveal complexities regarding word-level PPL interpretation across tokenizations and integration methods:

- Word-level PPL for LLMs must be interpreted carefully as it is on a different scale from subword-level PPL.

- Under word-level scoring, Qwen2 LLMs improve over recognition without an external LM even when its PPL is higher than the standard LM.

- Using newline as EOS reduces PPL substantially without changing WER, suggesting that EOS handling can strongly affect intrinsic PPL without changing the recognition ranking.

- Fine-tuning with the ASR SPM vocabulary gives the best LLM results, matching the Transformer LM and improving further to 3.65% WER for Qwen2-1.5B-FT-SPM.

The paper concludes that while the PPL-WER relation remains informative, it is not always a single global loglog trend, and the behavior depends on whether internal language modeling capacity is considered or how LLMs are integrated into the recognition pipeline. The findings emphasize that for AED systems, ILM subtraction is an important factor when studying external LM quality. Finally, for LLMs, the best results are achieved when fine-tuning with the ASR SentencePiece (SPM) vocabulary is used. The relation between LM perplexity and WER for modern end-to-end ASR is revisited, showing that external LMs still help, especially with restricted encoder context. The relation remains informative, but is not always a single global loglog trend: several setups show a low-PPL region with larger WER sensitivity and a high-PPL region with smaller slope. For Connectionist temporal classification (CTC), external LMs still help, especially with restricted encoder context. For attentionbased encoder-decoder (AED) ASR, the internal language model (ILM) affects how external LM quality maps to WER, and ILM subtraction increases the low-PPL slope. Finally, the large language model (LLM) experiments show that wordlevel PPL is harder to interpret across tokenizations and integration methods: under word-level scoring, Qwen2 LLMs improve over recognition without an external LM even when its PPL is higher than the standard LM, newline-based endof-sentence (EOS) handling changes PPL without changing WER, and Qwen2 LLMs with the ASR SentencePiece (SPM) vocabulary give the best LLM results. The relation between language model (LM) perplexity and automatic speech recognition (ASR) word error rate for modern end-to-end ASR is revisited, showing that external LMs still help, especially with restricted encoder context. The relation remains informative, but is not always a single global loglog trend: several setups show a low-PPL region with larger WER sensitivity and a high-PPL region with smaller slope. For Connectionist temporal classification (CTC), external LMs still help, especially with restricted encoder context. For attentionbased encoder-decoder (AED) ASR, the internal language model (ILM) affects how external LM quality maps to WER, and ILM subtraction increases the low-PPL slope. Finally, the large language model (LLM) experiments show that wordlevel PPL is harder to interpret across tokenizations and integration methods: under word-level scoring, Qwen2 LLMs improve over recognition without an external LM even when its PPL is higher than the standard LM, newline-based endof-sentence (EOS) handling changes PPL without changing WER, and Qwen2 LLMs with the ASR SentencePiece (SPM) vocabulary give the best LLM results. The relation between language model (LM) perplexity and automatic speech recognition (ASR) word error rate for modern end-to-end ASR is revisited, showing that external LMs still help, especially with restricted encoder context. The relation remains informative, but is not always a single global loglog trend: several setups show a low-PPL region with larger WER sensitivity and a high-PPL region with smaller slope.

Improvements for AI systems

External LM integration in CTC ASR should be tuned using a log-linear score combining frame-level CTC labels and a sequence-level LM score, specifically using a scale lambda for the LM and mu for the frame-wise prior to model recognition. This allows for better control over how external language model quality impacts recognition performance.
For AED ASR systems, applying internal language model subtraction using an estimate like Mini-LSTM ILM during recognition is recommended because it increases the slope from 0.19 to 0.29 in the low-perplexity regime, thereby strengthening the correlation between perplexity and WER and increasing external LM gain.
Encoder context length should be strategically managed; results show that the external LM becomes more important when the encoder has less acoustic context, suggesting that for short context lengths, the ASR system can compensate for missing acoustic information using external language model knowledge during search.
When evaluating Large Language Models (LLMs) as external LMs, word-level perplexity should be interpreted carefully across different tokenizations; specifically, fine-tuning LLMs with the ASR SentencePiece vocabulary yields the best results because Qwen2-0.5B-FT-SPM matches the Transformer LM (3.76% vs. 3.75%) in WER.
For AED systems, recognition can leverage internal language modeling by subtracting an estimate of the ILM, which is crucial because the ILM acts as a strong prior that can reduce the gain from external LMs and also weaken the correlation between perplexity and WER.