Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605.13087 · cs.CL, cs.AI · Submitted 2026-05-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition".

Jane: Here's the gist of the paper. The gist: Reverse multi-stage fine-tuning (R-MFT) enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts for spontaneous Indic speech by employing a high initial learning rate and a hard-to-easy curriculum.

Tom: Vividh-ASR Benchmark Vividh-ASR is introduced as a complexity-stratified benchmark designed to diagnose studio-bias in Indic speech recognition It stratifies evaluation by acoustic complexity across four tiers: studio (Tier A), broadcast (Tier B), spontaneous (Tier C), and synthetic noise (Tier D) Tier A consists of "Scripted, read speech in controlled environments" Tier B involves "Read speech from news broadcasts" Tier C represents "Crowdsourced, unscripted recordings with disfluencies, varying prosody, background noise, and non-professional hardware" Tier D is defined as "Tier A audio augmented with synthetic noise profiles (babble, music, environmental)" The data distribution is intentionally weighted toward Tier C, reflecting the focus on closing the spontaneous-speech performance gap.

Jane: Optimization Dynamics and Curriculum Study The authors systematically decoupled the effects of learning-rate timing and curriculum direction through a controlled 2×2 factorial study The two axes tested were Learning Rate (LR) Timing, comparing a decreasing schedule against an increasing schedule, and Curriculum Direction, comparing an easy-to-hard ordering against a hard-to-easy ordering.

Tom: Key findings from this study indicate that "early large parameter updates improve global WER by ∼12 absolute points" while a "hard-to-easy curriculum adds gains for spontaneous speech".

Title and authors: Jane: So, we're looking at this paper now called Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition. Tom, what's the basic idea here for a listener who isn't deep in machine learning?

Tom: Basically, they’re talking about how fine-tuning big speech models like Whisper works when you try to use them for languages that aren't super common. The authors point out that if you just fine-tune on read speech, it actually makes the model worse when it tries to understand spontaneous speech, which is how people actually talk in real life.

Lu: They introduce this benchmark called Vividh-ASR to diagnose exactly where models fail by stratifying evaluation across four tiers: studio, broadcast, spontaneous, and synthetic noise. It's meant to isolate the problem of studio bias in Indic speech recognition across these different levels of acoustic difficulty <ref:2605.13087#pg1>.

Meng: So it’s not just one test; it breaks down performance based on how noisy or complex the audio actually is—from controlled recordings to real crowds with background noise <ref:2605.13087#pg1>. It lets them see if a model works for casual conversation or if it only works well in perfect studio settings.

Lalam: I see this as a way to stop people from just assuming that because a model does okay on read speech, it's actually ready for spontaneous audio too <ref:2605.13087#pg2>.

The paper's summary: Tom: The core finding they present is that simply using a standard fine-tuning approach doesn't fix this mismatch; you need to change *how* you tune the model. They studied learning rate timing and curriculum ordering to see what actually helps.

Jane: What they found was pretty concrete: starting with a high learning rate helps improve the overall Word Error Rate by about twelve absolute points <ref:2605.13087#pg1> <ref:2605.13087#pg1>. And they also found that a hard-to-easy order for the data actually adds gains specifically for spontaneous speech <ref:2605.13087#pg1>.

Lu: This leads them to propose something called Reverse Multi-Stage Fine-Tuning, or R-MFT, which is a training recipe designed to be parameter efficient. It uses three stages and a specific learning rate schedule that pairs the best conditions together <ref:2605.13087#pg3>.

Meng: From an engineering standpoint, they are trying to get a smaller two hundred forty-four million Whisper model to perform better than a much larger seven hundred sixty-nine million baseline by using this R-MFT recipe <ref:2605.13087#pg1>. That parameter efficiency is what’s really interesting for us at scale.

Lalam: It seems like the idea is that you need to break out of the model's initial pre-trained state with a big push before you can get it to learn the harder stuff <ref:2605.13087#pg2>.

The paper's improvements: Tom: So what are they suggesting we actually *do* differently? They are suggesting this R-MFT recipe, which is essentially pairing a high initial learning rate with a hard-to-easy curriculum for the first stage of training <ref:2605.13087#pg3>.

Jane: Think about it this way: you start aggressive to get the model moving past its starting point, then you move through different data difficulties in an order that gradually gets harder <ref:2605.13087#pg1>. It’s about controlling the learning process itself rather than just throwing more data at it randomly.

Lu: The paper also shows how this works internally using Centered Kernel Alignment and Singular Value Decomposition to prove that these effective schedules concentrate the adaptation in the decoder part of the model while keeping the encoder's acoustic geometry stable <ref:2605.13087#pg1>.

Meng: That’s a big deal for deployment because it means we aren't destroying all the knowledge in the core encoder; we’re just adjusting what processes audio into text, which makes it much more efficient to fine-tune and maintain.

Lalam: So the improvement is not just about getting better scores on a test set; it’s about finding a training path that respects how speech models are built internally <ref:2605.13087#pg1>.

Conclusion: Jane: So, to wrap up on Vividh-ASR, the main implication is that for Indic ASR, raw model scaling isn't always the answer; controlling the optimization trajectory with R-MFT can actually outperform simply using a bigger model <ref:2605.13087#pg1>.

Tom: The key thing to remember is that aggressive early learning rates are dominant; they yield about twelve absolute points in Word Error Rate improvement, and later high-energy updates don't help if you start too conservatively <ref:2605.13087#pg3> <ref:2605.13087#pg1>.

Lu: It confirms that a hard-data first optimization trajectory is more effective than raw model scaling for getting robust Indic ASR results <ref:2605.13087#pg1>.

Meng: For deployment, this means we can potentially use smaller models like the 244M version because the R-MFT recipe makes them match or exceed the performance of those larger 769M counterparts on spontaneous speech <ref:2605.13087#pg1> <ref:2605.13087#pg1>.

Lalam: I think what matters most is that adaptation efficiency is governed by two things: timing of large parameter updates and ordering of acoustic complexity <ref:2605.13087#pg1>.

Tom: Exactly, so we’re seeing a pattern here where the way you introduce that initial high learning rate sets up the whole process for success or failure <ref:2605.13087#pg3>.

Jane: That gives us a clear path forward for anyone working on low-resource languages, showing them exactly what kind of training recipe makes sense to test first <ref:2605.13087#pg1>.

Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil

Adalat AI, India

cs.CL, cs.AI

Submitted: 2026-05-13

Updated: 2026-06-29

Comments: Accepted at Interspeech 2026

Journal ref: Proc. Interspeech 2026, 3258-3262

DOI: 10.21437/Interspeech.2026-3408

Code: https://github.com/vasistalodagala/whisper-finetune

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 90/100

The gist: The gist: Reverse multi-stage fine-tuning (R-MFT) enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts for spontaneous Indic speech by

Key concepts

Vividh-ASR Benchmark: This is a new testing system that sorts speech data into four complexity levels: studio, broadcast, spontaneous, and synthetic noise. It helps identify exactly where speech recognition models struggle across different real-world conditions.
Reverse Multi-Stage Fine-Tuning (R-MFT): This is a specific training recipe. It starts with high learning rates on the hardest data (spontaneous) first, then moves to easier data. This strategy helps the model adapt efficiently while keeping its core knowledge intact.
Centered Kernel Alignment (CKA) and SVD: These are mathematical tools used to look inside the AI model's internal workings. They show that successful fine-tuning causes changes only in the decoder part of the model, leaving the encoder's foundational acoustic understanding unchanged.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Adaptation — 1× in this paper
Decoder — 1× in this paper · explained in 2 episodes
Low-Resource Languages — 1× in this paper · explained in 4 episodes

Summary

The gist: Reverse multi-stage fine-tuning (R-MFT) enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts for spontaneous Indic speech by employing a high initial learning rate and a hard-to-easy curriculum.

Vividh-ASR Benchmark

Vividh-ASR is introduced as a complexity-stratified benchmark designed to diagnose studio-bias in Indic speech recognition It stratifies evaluation by acoustic complexity across four tiers: studio (Tier A), broadcast (Tier B), spontaneous (Tier C), and synthetic noise (Tier D) Tier A consists of Scripted, read speech in controlled environments Tier B involves Read speech from news broadcasts Tier C represents Crowdsourced, unscripted recordings with disfluencies, varying prosody, background noise, and non-professional hardware Tier D is defined as Tier A audio augmented with synthetic noise profiles (babble, music, environmental) The data distribution is intentionally weighted toward Tier C, reflecting the focus on closing the spontaneous-speech performance gap.

Optimization Dynamics and Curriculum Study

The authors systematically decoupled the effects of learning-rate timing and curriculum direction through a controlled 2×2 factorial study The two axes tested were Learning Rate (LR) Timing, comparing a decreasing schedule against an increasing schedule, and Curriculum Direction, comparing an easy-to-hard ordering against a hard-to-easy ordering. Key findings from this study indicate that early large parameter updates improve global WER by ∼12 absolute points while a hard-to-easy curriculum adds gains for spontaneous speech. The R-MFT recipe pairs the optimal conditions, consisting of a high initial learning rate to break out of the pre-trained basin, coupled with a spontaneous-first (hard-to-easy) curriculum.

Reverse Multi-Stage Fine-Tuning (R-MFT)

The R-MFT training recipe consists of three stages designed to maximize adaptation efficiency 1. Stage 1 involves Tier C data with a LR = 2e−4, where the highest-plasticity phase encounters the highest-complexity data, explicitly building robustness to disfluencies, noise, and varying prosody 2. Stage 2 uses Tier B data with LR = 1e−4, which is used to Refine temporal modeling for rapid speech 3. Stage 3 is a consolidation phase using a LR = 1e−5 and a 1:1 mixture by duration of Tier A and Tier C to act as a regularizer. This recipe allows the model to achieve remarkable parameter efficiency, with the 244M R-MFT (Small) model achieving global WERs that outperforms the much larger 769M Single-stage low-LR baseline by 33.38 and 3.84 absolute points respectively.

Representational Analysis of Adaptation

Representational analysis via CKA and SVD reveals that effective schedules concentrate adaptation in the decoder while largely preserving the pre-trained encoder’s acoustic geometry This structural asymmetry is confirmed by examining internal model representations using centered kernel alignment (CKA) [8] and singular value decomposition (SVD). Table 5 shows that successful adaptation is characterized by substantial parameter displacement concentrated almost exclusively in the decoder while the encoder’s pre-trained geometry remains invariant. The analysis of spectral signatures shows that IndicWhisper, fine-tuned predominantly on read and broadcast speech, severely disrupts the pre-trained encoder’s geometry, dropping CKA to 0.775. In contrast, the R-MFT schedule achieves superior adaptation entirely through decoder realignment, leaving the foundational acoustic robustness of the encoder intact.

Conclusion and Future Directions

The work demonstrates that aggressive early learning rates are the dominant factor, yielding ∼ 12 absolute WER points and that later high-energy updates cannot recover performance lost to a conservative, low-LR initialization. The R-MFT recipe proves that a hard-data first optimization trajectory is more effective than raw model scaling for robust Indic ASR Future work will extend Vividh-ASR to additional languages and investigate the efficacy of selective encoder freezing as a regularization strategy to further mitigate studio-bias. The findings suggest that adaptation efficiency is governed by two interacting factors: the timing of large parameter updates and the ordering of acoustic complexity. The paper releases both the benchmark and models for community use.

How it works

The primary mechanism for improving performance involves manipulating optimization plasticity through learning rate timing and data ordering Specifically, the R-MFT recipe utilizes a hard-to-easy curriculum paired with a high initial learning rate to escape the pre-trained loss basin. This strategy is effective because it provides the necessary gradient energy for the decoder to escape the pre-trained basin while the encoder remains anchored to its robust acoustic features.

Key Contributions

The paper makes two primary contributions: 1) Vividh-ASR Benchmark, a diagnostic benchmark that stratifies evaluation by acoustic complexity: studio (Tier A), broadcast (Tier B), spontaneous (Tier C), and synthetic noise (Tier D) 2) Reverse multi-stage fine-tuning (R-MFT), a training recipe that pairs the optimal conditions, enabling a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts. The analysis using CKA and SVD confirms that this method adapts the decoder to target-language phonotactics while preserving the encoder’s robust acoustic geometry.

Performance Comparison

Table 4 summarizes the performance across Vividh-ASR tiers for Malayalam (Mal) and Hindi (Hi) The R-MFT recipe yields results such as 44.41% (Mal) and 21.41% (Hi) global WER This represents a significant improvement over the baseline, showing that a hard-data first optimization trajectory is more effective than raw model scaling for robust Indic ASR. The study concludes that high-magnitude updates must be applied at the outset to escape the pre-trained prior; delayed high-energy updates are insufficient to exit sub-optimal representational basins formed during a conservative initialization.

Related Work Context

The research builds upon existing work in Indic ASR corpora and benchmarks, such as Kathbath [9] and Indic Voices [11] It also engages with curriculum learning for ASR, studying how curriculum direction interacts with optimization schedules during fine-tuning. Furthermore, the study bridges a gap by examining how fine-tuning schedules physically reshape internal representations using centered kernel alignment (CKA) [8] and singular value decomposition (SVD). The research also considers the structural asymmetry between encoder and decoder adaptation, noting that R-MFT achieves this by concentrating displacement in the decoder while preserving the encoder's geometry <ref:2605.

Improvements for AI systems

The improved system will implement Reverse Multi-Stage Fine-Tuning (R-MFT), a training recipe that pairs spontaneous-first data ordering with high initial learning rates. This enables a parameter efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts on spontaneous Indic speech.

The system will utilize Vividh-ASR, a complexity-stratified benchmark for Hindi and Malayalam across four tiers: studio, broadcast, spontaneous, and synthetic noise. This allows for a diagnostic evaluation that isolates precisely where models fail along the complexity axis, moving beyond domain-based benchmarks.

The improved system will incorporate internal representation analysis using Centered Kernel Alignment (CKA) and Singular Value Decomposition (SVD) during fine-tuning. This ensures that adaptation concentrate[s] in the decoder, preserving the pre-trained encoder’s acoustic geometry, preventing the loss of generalized acoustic invariance.

Abstract

Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance. To diagnose this mismatch, we introduce Vividh-ASR, a complexity-stratified benchmark for Hindi and Malayalam across four tiers: studio, broadcast, spontaneous, and synthetic noise. Through a controlled study of learning-rate timing and curriculum ordering, we find that early large parameter updates improve global WER by 12 absolute points, while a hard-to-easy curriculum adds gains for spontaneous speech. These findings motivate reverse multi-stage fine-tuning (R-MFT), a training recipe that enables a parameter-efficient 244M Whisper model to match or exceed conventionally fine-tuned 769M counterparts. Representational analysis via CKA and SVD reveals effective schedules concentrate adaptation in the decoder, preserving the pre-trained encoder's acoustic geometry. We release the benchmark and models.