SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR".

Jane: Here's the gist of the paper. The gist The SCRIBE framework introduces a diagnostic evaluation system that decomposes Automatic Speech Recognition (ASR) errors into lexical, punctuation, numeral, and domain-entity rates to provide actionable development signals for Indic languages.

Tom: The gist SCRIBE is a diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection.

Tom: The framework transforms reference R and hypothesis H into typed tokens (wi, ti), where ti ∈ the set of lexeme, numeral, punctuation, or domain-entity; this is done by treating standard punctuation and Indic-specific marks as independent tokens while preserving those within numerals and compound words to maintain lexical integrity.

Title and authors: Tom: So, what’s the actual title of this work we’re hearing about today? It’s called SCRIBE: Diagnostic Evaluation and Rich Transcription Models for Indic ASR.

Jane: That title tells you a lot; it suggests they are not just building another transcription model, but they are focused on how to evaluate them in a way that actually helps developers improve their systems.

Lu: They introduce this diagnostic framework that provides categorical error decomposition into lexical, punctuation, numeral, and domain-entity rates through sandhi-tolerant alignment with domain vocabulary injection.

Meng: That sandhi-aware alignment engine sounds complicated but it seems to be crucial for Indic languages where those language merges can really mess up standard recognition systems.

Lalam: When they resolve those tricky one to two or two to one mappings, they score them based on phonetic plausibility, which is a smart way to handle those specific language phenomena.

Tom: The implication here is that instead of just getting a number for how bad the system is overall, you get actionable data on exactly *what* went wrong and where to start fixing it.

Jane: Exactly. It shifts the conversation from just knowing *that* the system failed to understanding precisely *why* it failed, like fixing a misrecognized domain term versus just inserting a comma.

Lu: They also designed this whole cycle, Figure one shows, as a diagnostic-led development cycle for Indic rich transcription.

Meng: From a deployment standpoint, isolating domain entity errors means we know exactly which vocabulary injection or fine-tuning step in the domain data is causing the failure, which is much better than just getting a high aggregate number.

Lalam: I’m confident this framework works because it aligns with human judgment; they saw a correlation of zero point three six to zero point nine two between SCRIBE’s error rates and what experts prioritize.

The paper's summary: Tom: So, let’s talk about the actual results of the SCRIBE paper. They start by showing how they created rich transcription data using an LLM curation pipeline that uses language-specific prompts to transform public Indic speech corpora into something much richer.

Jane: Then they fine-tune the ASR model from a pre-trained Whisper architecture through three distinct stages: diversity adaptation across acoustic conditions, pace and style robustness, and precision tuning with near perfect well articulated speech.

Lu: They also introduce two new benchmarks for evaluation: FLEURS-RO for general rich orthography and IN22-Legal for domain-specific evaluation where ground truth transcripts were prepared directly in rich transcription format by legal domain annotators.

Meng: And they show that SCRIBE’s categorical metrics align with human judgment, with a correlation of zero point three six to zero point nine two, which is significantly better than the four point nine two for monolithic word error rate.

Lalam: They found a critical divergence where genuine lexical failure ERlex accounts for only fifteen point nine six percent in the Malayalam Legal set despite a high monolithic word error rate of forty-four point five two percent.

Tom: That divergence shows that core acoustic-phonetic reliability is nearly three times higher than the monolithic scalar suggests, and that much of the reported Indic word error rate is an artifact of morphological structure rather than just acoustic misrecognition.

Jane: It means we can finally see that core acoustic reliability is actually much higher than the standard scalars suggest, and that a lot of reported errors are just artifacts of how Indic language works grammatically.

Lu: And they’re showing that this framework resolves valid morphological merges reduces reported error inflation by roughly thirty percent relative in both Malayalam and Kannada.

The paper's improvements: Tom: So, what are the specific improvements they point out with SCRIBE? They focus on how the system can be used practically for fixing errors.

Jane: First, SCRIBE provides a diagnostic complement to monolithic word error rate wherever ASR serves as a scribe by enabling targeted remediation through sandhi-tolerant alignment and categorical decomposition.

Lu: It’s summarized in the conclusion that they release SCRIBE alongside their curation recipe, benchmarks, and openweight models for Hindi, Malayalam, and Kannada to enable development of ASR systems meeting correction thresholds for professional dictation.

Meng: The main bottleneck they identified is punctuation as ERpunc remains the biggest challenge, especially in Dravidian languages, suggesting prosodic modeling might be a good next development target.

Lalam: And they showed that near saturation of numeral formatting, with ERnum under one percent, shows a seventy-five to ninety-six percent relative reduction compared to the best baseline across all benchmarks.

Tom: So we’re looking at SCRIBE as a way to get more precise control over ASR systems in Indic languages by focusing on these specific error modes instead of just the big number.

Jane: It’s a really useful tool for developers who need to know exactly what kind of error they are facing when they look at the output during their work.

Conclusion: Tom: So what we’re hearing is that SCRIBE gives us a diagnostic tool for Indic ASR systems that breaks down errors into specific categories like lexical, punctuation, numeral, and domain entity rates instead of just a single word error rate.

Jane: Right, Tom. It’s moving from just knowing *that* the system failed to knowing *why* it failed—like fixing a misrecognized domain term versus just inserting a comma.

Lu: The methodology uses sandhi-tolerant alignment, which means it handles those tricky language merges and splits that usually confuse standard systems, and it ties that alignment into a scoring function that penalizes category clashes with negative three point zero.

Lalam: I’m sure the direction is solid; SCRIBE provides that diagnostic complement to monolithic word error rate wherever ASR serves as a scribe.

Tom: Yeah, so we’re looking at SCRIBE as a way to get more precise control over ASR systems in Indic languages by focusing on these specific error modes instead of just the big number.

Lu: We’re excited to see how this diagnostic approach helps us tackle those complex morphophonological challenges inherent in Indic speech processing moving forward.

Meng: I'm still cautious about deploying this at scale without running it on real traffic because those benchmarks are controlled, but the evidence for domain entity error reduction is compelling.

Lalam: I’m sure the direction is solid; SCRIBE provides that diagnostic complement to monolithic word error rate wherever ASR serves as a scribe.