SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR

arXiv:2605.20712 · cs.CL, cs.AI · Submitted 2026-05-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR".

Jane: Here's the gist of the paper. The gist The SCRIBE framework introduces a diagnostic evaluation system that decomposes Automatic Speech Recognition (ASR) errors into lexical, punctuation, numeral, and domain-entity rates to provide actionable development signals for Indic languages.

Tom: The gist SCRIBE is a diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection.

Tom: The framework transforms reference R and hypothesis H into typed tokens (wi, ti), where ti ∈ the set of lexeme, numeral, punctuation, or domain-entity; this is done by treating standard punctuation and Indic-specific marks as independent tokens while preserving those within numerals and compound words to maintain lexical integrity.

Title and authors: Tom: So, what’s the actual title of this work we’re hearing about today? It’s called SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR.

Jane: That title tells you a lot; it suggests they are not just building another transcription model, but they are focused on how to evaluate them in a way that actually helps developers improve their systems.

Lu: They introduce this diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection.

Meng: That sandhi-aware alignment engine sounds complicated but it seems to be crucial for Indic languages where those language merges can really mess up standard recognition systems.

Lalam: When they resolve those tricky one to two or two to one mappings, they score them based on phonetic plausibility, which is a smart way to handle those specific language phenomena.

Tom: The implication here is that instead of just getting a number for how bad the system is overall, you get actionable data on exactly *what* went wrong and where to start fixing it.

Jane: Exactly. It shifts the conversation from just knowing *that* the system failed to understanding precisely *why* it failed, like fixing a misrecognized domain term versus just inserting a comma.

Lu: They also designed this whole cycle, Figure one shows, as a diagnostic-led development cycle for Indic rich transcription.

Meng: From a deployment standpoint, isolating domain entity errors means we know exactly which vocabulary injection or fine-tuning step in the domain data is causing the failure, which is much better than just getting a high aggregate number.

Lalam: I’m confident this framework works because it aligns with human judgment; they saw a correlation of zero point three six to zero point nine two between SCRIBE’s error rates and what experts prioritize.

The paper's summary: Tom: So, let’s talk about the actual results of the SCRIBE paper. They start by showing how they created rich transcription data using an LLM curation pipeline that uses language-specific prompts to transform public Indic speech corpora into something much richer.

Jane: Then they fine-tune the ASR model from a pre-trained Whisper architecture through three distinct stages: diversity adaptation across acoustic conditions, pace and style robustness, and precision tuning with near perfect well articulated speech.

Lu: They also introduce two new benchmarks for evaluation: FLEURS-RO for general rich orthography and IN22-Legal for domain-specific evaluation where ground truth transcripts were prepared directly in rich transcription format by legal domain annotators.

Meng: And they show that SCRIBE’s categorical metrics align with human judgment, with a correlation of zero point three six to zero point nine two, which is significantly better than the four point nine two for monolithic word error rate.

Lalam: They found a critical divergence where genuine lexical failure ERlex accounts for only fifteen point nine six percent in the Malayalam Legal set despite a high monolithic word error rate of forty-four point five two percent.

Tom: That divergence shows that core acoustic-phonetic reliability is nearly three times higher than the monolithic scalar suggests, and that much of the reported Indic word error rate is an artifact of morphological structure rather than just acoustic misrecognition.

Jane: It means we can finally see that core acoustic reliability is actually much higher than the standard scalars suggest, and that a lot of reported errors are just artifacts of how Indic language works grammatically.

Lu: And they’re showing that this framework resolves valid morphological merges reduces reported error inflation by roughly thirty percent relative in both Malayalam and Kannada.

The paper's improvements: Tom: So, what are the specific improvements they point out with SCRIBE? They focus on how the system can be used practically for fixing errors.

Jane: First, SCRIBE provides a diagnostic complement to monolithic word error rate wherever ASR serves as a scribe by enabling targeted remediation through sandhi-tolerant alignment and categorical decomposition.

Lu: It’s summarized in the conclusion that they release SCRIBE alongside their curation recipe, benchmarks, and openweight models for Hindi, Malayalam, and Kannada to enable development of ASR systems meeting correction thresholds for professional dictation.

Meng: The main bottleneck they identified is punctuation as ERpunc remains the biggest challenge, especially in Dravidian languages, suggesting prosodic modeling might be a good next development target.

Lalam: And they showed that near saturation of numeral formatting, with ERnum under one percent, shows a seventy-five to ninety-six percent relative reduction compared to the best baseline across all benchmarks.

Tom: So we’re looking at SCRIBE as a way to get more precise control over ASR systems in Indic languages by focusing on these specific error modes instead of just the big number.

Jane: It’s a really useful tool for developers who need to know exactly what kind of error they are facing when they look at the output during their work.

Conclusion: Tom: So what we’re hearing is that SCRIBE gives us a diagnostic tool for Indic ASR systems that breaks down errors into specific categories like lexical, punctuation, numeral, and domain entity rates instead of just a single word error rate.

Jane: Right, Tom. It’s moving from just knowing *that* the system failed to knowing *why* it failed—like fixing a misrecognized domain term versus just inserting a comma.

Lu: The methodology uses sandhi-tolerant alignment, which means it handles those tricky language merges and splits that usually confuse standard systems, and it ties that alignment into a scoring function that penalizes category clashes with negative three point zero.

Lalam: I’m sure the direction is solid; SCRIBE provides that diagnostic complement to monolithic word error rate wherever ASR serves as a scribe.

Tom: Yeah, so we’re looking at SCRIBE as a way to get more precise control over ASR systems in Indic languages by focusing on these specific error modes instead of just the big number.

Lu: We’re excited to see how this diagnostic approach helps us tackle those complex morphophonological challenges inherent in Indic speech processing moving forward.

Meng: I'm still cautious about deploying this at scale without running it on real traffic because those benchmarks are controlled, but the evidence for domain entity error reduction is compelling.

Lalam: I’m sure the direction is solid; SCRIBE provides that diagnostic complement to monolithic word error rate wherever ASR serves as a scribe.

Kavya Manohar, Arghya Bhattacharya, Kush Juvekar, Kumarmanas Nethil

Adalat AI, India

cs.CL, cs.AI

Submitted: 2026-05-20

Updated: 2026-06-29

Comments: Accepted at Interspeech 2026

Journal ref: Proc. Interspeech 2026, 4773-4778

DOI: 10.21437/Interspeech.2026-3436

Code: https://github.com/adalat-ai-tech/scribe-eval2https:

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 92/100

The gist: The gist The SCRIBE framework introduces a diagnostic evaluation system that decomposes Automatic Speech Recognition (ASR) errors into lexical, punctuation, numeral, and domain-entity rates to

Key concepts

Diagnostic Error Vector E
This vector decomposes ASR failures into four distinct categories: lexical (words), punctuation, numerals, and domain-specific entities. Instead of one total error score, this allows developers to pinpoint exactly where the system is failing—for example, distinguishing between misrecognizing a word and incorrectly placing a comma.
Sandhi-Tolerant Alignment
This is an advanced alignment technique that handles specific phonetic rules in Indic languages called sandhi. It recognizes that words can merge or split phonetically during speech, allowing the system to correctly map spoken sounds to written text even when standard algorithms fail due to these linguistic features.
Categorical Error Rates (ERt)
These are the specific error rates for each category (lexical, punctuation, numeral, etc.). The framework calculates these using a combined denominator that prevents sparse errors from skewing the results. This provides a detailed view of system performance across different types of transcription mistakes.
LLM Curation Pipeline
This is a process using Large Language Models to transform public speech data into high-quality training text. It applies language-specific prompts and fine-tuning stages to create rich, domain-specific transcripts, which helps the ASR model learn complex vocabulary and formatting conventions.

Terminology

Summary

The gist The SCRIBE framework introduces a diagnostic evaluation system that decomposes Automatic Speech Recognition (ASR) errors into lexical, punctuation, numeral, and domain-entity rates to provide actionable development signals for Indic languages.

The gist

SCRIBE is a diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection.

How it works

  1. The framework transforms reference R and hypothesis H into typed tokens (wi, ti), where ti ∈ the set of lexeme, numeral, punctuation, or domain-entity; this is done by treating standard punctuation and Indic-specific marks as independent tokens while preserving those within numerals and compound words to maintain lexical integrity.

  2. The sandhi-aware alignment engine seeks the alignment maximizing a total score dp[i][j], calculated via extended dynamic programming in Equation 1, which accounts for insertions, deletions, standard 1:1 substitutions, and Sandhi-motivated 1:2 (split) and 2:1 (merge) mappings.

  3. The scoring function S(ri, hj) anchors the alignment on exact matches with a score of α = +4.0 while buffering against acoustic near-misses common in Indic scripts, and a category-clash penalty β = −3.0 is applied if tR i ̸= tH j; for same-category substitutions, a Levenshtein-buffered penalty δ = −1.5 − (0.2 · d) is employed where d is the character distance between ri and hj, with a Unicodelevel distance of d ≤ 2 optimally capturing minor orthographic variations without triggering cascading deletion-insertion pairs typical of standard WER evaluation.

  4. Sandhi scores, Σ, resolve 1:2 or 2:1 mappings by validating phonetic plausibility, scoring this as Σ = α+σ−d(bsplit, bmid)/s, where σ = −0.5 is the sandhi penalty; a transition is valid if the fused form s matches the prefix of w1 and suffix of w2, and if the boundary distance d > 2, the transition is invalidated per Indic morphophonological rules.

The gist

SCRIBE outputs a diagnostic error vector E = [ERlex, ERpunc, ERnum, ERent], which decomposes failures into lexical, punctuation, numeral, and domain-specific entity error rates respectively.

How it works

  1. SCRIBE aggregates errors into a diagnostic vector E where each component maps to a specific remediation strategy; each rate uses a combined denominator Ncomb = ∑ t total[t]: ERt = (sub[t]+ins[t]+del[t])/Ncomb, ensuring sparse-category failures do not produce misleadingly high rates and that their sum, SCRIBEWER (WERS = ∑ t ERt), serves as a single global rate with sandhi inflation removed.

  2. The framework utilizes an LLM curation pipeline to transform public Indic speech corpora into rich transcription by applying language-specific prompts, and then fine-tunes the ASR model from a pre-trained Whisper architecture in three stages: diversity adaptation across acoustic conditions, pace and style robustness, and precision tuning with near perfect well articulated speech.

  3. The evaluation involves two new benchmarks: FLEURS-RO for general rich orthography derived from the FLEURS multilingual test set, and IN22-Legal for domain-specific evaluation derived from IN22, where ground-truth transcripts were prepared directly in rich transcription format by legal-domain annotators to test generalization to high-stakes specialized vocabulary.

The gist

SCRIBE’s categorical metrics align robustly with human judgment (ρ=0.36–0.92), significantly outperforming monolithic WER (ρ≤0.49).

How it works

  1. The validation study designs a correlation study where human ratings serve as the ground truth against which both SCRIBE and WER are measured, computing Spearman ρ between SCRIBE’s categorical error rates (ERlex, ERnum, ERpunc) and their corresponding human dimensions (S1, S2, S3), demonstrating that experts prioritize functional categories—specifically punctuation and numerals—that global WER treats as noise.

  2. The analysis reveals a critical divergence where genuine lexical failure (ERlex) accounts for only 15.96% in the Malayalam Legal set despite a high monolithic WER of 44.52%, proving that core acoustic-phonetic reliability is nearly 3× higher than the monolithic scalar suggests, and that a substantial portion of reported Indic WER is an artifact of morphological structure rather than acoustic misrecognition.

  3. The diagnostic value extends to active development via a feedback loop where early iterations exhibited systematic over-punctuation bias entirely invisible in aggregate WER, which SCRIBE’s ERpunc decomposition isolated to segments with legacy verbatim corpora containing extremely short sequences with misleading terminal punctuation, enabling targeted filtering and refined LLM curation prompts.

The gist

SCRIBE proves that Indic ASR systems are more acoustically reliable than standard scalars suggest.

How it works

  1. The framework’s diagnostic analysis reveals that resolving valid morphological merges reduces reported error inflation by roughly 30% relative in both Malayalam and Kannada, and near-saturation of numeral formatting (ERnum < 1%) shows a 75–96% relative reduction compared to the best baseline across all benchmarks, indicating the effectiveness of the LLM curation pipeline in producing training data whose formatting conventions extend to unseen domains.

  2. The primary bottleneck identified is punctuation as ERpunc remains the main challenge, particularly in Dravidian languages, pointing toward prosodic modeling as a next development target; this is evidenced by Malayalam Legal reporting 12.12% vs. Hindi’s 6.73% for ERpunc in Table 1.

The SCRIBE framework provides a diagnostic complement to monolithic WER wherever ASR serves as a scribe, enabling targeted remediation for rich transcription tasks through sandhi-tolerant alignment and categorical decomposition; this is summarized in the conclusion of the paper. The authors release SCRIBE alongside their curation recipe, benchmarks, and open-weight models to enable development of ASR systems meeting the correction thresholds required for professional dictation.

<ref:2605.

Improvements for AI systems

  1. ASR systems can be evaluated by a diagnostic error vector E = [ERlex, ERpunc, ERnum, ERent] instead of a single scalar WER. This allows for targeted remediation of specific failure modes like fixing a misrecognized domain term versus inserting a comma.

  2. The improved system can achieve high accuracy in complex Indic languages by employing sandhi-tolerant alignment that accounts for 1:2 (split) and 2:1 (merge) mappings, which the paper shows reduces reported error inflation by roughly 30% relative to standard WER.

  3. A structured annotation schema and validation procedure can be used to ensure categorical ASR metrics align with expert judgment, validating that SCRIBE aligns with expert judgment where WER does not.

  4. The system can generate a reproducible recipe for Indic rich transcription using an LLM-based data curation pipeline, enabling the creation of training data that includes domain entity injection and formatting conventions.

  5. A model can be fine-tuned to show near saturation in numeral formatting (ERnum < 1%) and low domain entity error (ERent below 2%), demonstrating that acoustic learning from general corpora transfers effectively to specialized vocabulary.

  6. The system can provide an active development signal through a feedback loop, where categorical decomposition isolates specific regression, such as systematic over-punctuation bias, enabling targeted filtering of problematic data sequences.

Abstract

Automatic speech recognition replaces typing only when correction costs less than manual entry - a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma. Word error rate (WER) fails on two fronts: it collapses distinct error categories into a single scalar, and it structurally penalizes agglutinative languages where valid sandhi merges inflate scores. We introduce SCRIBE, a diagnostic framework offering categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates via sandhi-tolerant alignment with domain vocabulary injection. Human validation confirms SCRIBE aligns with expert judgment where WER does not. We release SCRIBE, an LLM curation pipeline, benchmarks, and open-weight rich transcription models for Hindi, Malayalam, and Kannada.

Sources

Related papers