From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "From Text to Voice".

Jane: Here's the gist of the paper. The gist: This study introduces a dataset-agnostic framework that converts existing text-based tool-calling benchmarks into controlled audio evaluations using Text-to-Speech (TTS) to provide a verifiable first-stage diagnostic for comparing cascade and end-to-end omni-modal models.

Tom: How it works The core of the framework is a systematic TTS conversion pipeline designed to create paired text–audio instances while preserving the original dataset annotations, tool schemas, and gold labels The pipeline utilizes three commercially available TTS models—Gemini-2.5-Flash-TTS3, Gemini-2.5-Pro-TTS4, and GPT-4o-Mini-TTS—to generate diverse audio queries with variations in speaker and environmental noise Speaker variation is achieved by selecting distinct voices from each TTS provider for both genders, such as Kore or Orus for Gemini voices, and Ash or Coral for GPT-4o-Mini Environmental noise is injected from the DEMAND dataset across various settings like cars, buses, and cafés at different signal-to-noise ratios (SNRs) to simulate controlled deployment conditions The benchmark is designed as an early-stage evaluation tool by evaluating each task in two aligned conditions: the text-only condition and the direct-audio condition This comparison allows for the estimation of performance loss caused by changing the input modality from text to speech while keeping the task, tool schema, and gold labels fixed Benchmark Conversion and Preservation The framework treats TTS-generated speech as a "controlled approximation of voice-based interaction" to isolate text–audio and cascade–omni performance gaps A key design choice is to evaluate each task in two aligned conditions, measuring the model’s downstream ability to use the correct tool when speech processing is not involved versus its ability to perform the same tool-calling task from speech This approach preserves verifiability by inheriting the source benchmark’s "tool schemas, gold labels, and evaluation protocol" The conversion is not merely a cheaper data-generation strategy; it preserves the source benchmark’s tool schemas, gold labels, and scoring protocol while testing the effect of changing only the input modality The framework is dataset-agnostic, meaning "any text-based tool-calling benchmark can be integrated into our framework without any architectural modification" For instance, the study employs two publicly available benchmarks: Confetti and When2Call Confetti measures function selection and parameter extraction accuracy, whereas When2Call assesses the decision-making capability required to determine whether to call a tool or not Model Evaluation and Diagnostics The study evaluates seven omni-modal models across four providers and multiple model tiers, including GPT-4o-Realtime, Gemini-3.1-Flash-Live, Qwen3-Omni, and Phi-4 (Abouelenin et al., 2025) Performance is measured across clean and noisy audio with diverse voices on both Confetti and When2Call datasets The results show that performance is "strongly model- and task-dependent, not determined by architecture alone" For example, Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9) A targeted analysis of failure cases demonstrates that "degradations most often reflect misunderstandings of argument values in the speech" This is evidenced by error decomposition on Confetti, where "argument-value errors are not explained by a single error type," with argument-value errors being the largest category for Gemini-3.1-Flash-Live (57.2%) and GPT-Realtime-1.5 (54.3%) Diagnostic Protocol Comparison The framework enables a diagnostic protocol by comparing three input settings: clean text, direct voice, and the ASR-to-text cascade Clean-text performance estimates the model’s tool-calling ceiling, while direct voice measures how well that capability transfers to speech input The comparison reveals that "the cost of moving from text to speech is model-dependent rather than a fixed property of omni-modal inference" For instance, the text-to-voice gap for Gemini-3.1-Flash-Live ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5 on Confetti Furthermore, the study assesses reference-free LLM-as-judge protocols for production evaluation, finding that "open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges" This suggests a path toward privacy-preserving evaluation Practical Implications for Deployment The results suggest that "voice-agent architecture should be selected through model- and task-specific evaluation rather than by assuming that cascade or omni-modal systems are uniformly better" Direct omni-modal models are attractive when the text-to-voice gap is small, whereas cascades remain safer when teams need vendor flexibility Clean-text performance provides the tool-calling ceiling, and the large drop under ambiguous reformulations shows that handling underspecified requests is as important as improving the audio stack Overall, this study provides a "verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora" The framework offers a "reusable recipe that enterprises can apply to proprietary text-based tool-calling data" This design has three advantages: it preserves gold labels and tool schema, creates paired text–audio instances for failure analysis, and can be applied to proprietary enterprise tool catalogs and text logs The framework provides a "verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora" The study concludes that audio-converted versions of trusted text benchmarks can complement purpose-built audio datasets.

Jane: Page 1 --- Md Tahmid Rahman Laskar tahmid.rahman@dialpad.com Dialpad Inc.

Tom: Xue-Yong Fu xue-yong@dialpad.com Dialpad Inc.

Title and authors: Tom: So we're looking at this paper called "From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents". It’s from Md Tahmid Rahman Laskar and his team, and they’re tackling the problem of how to test these new voice agents.

Jane: Exactly. The big issue is that most tool calling benchmarks are just text-based, but real customer support happens over voice, so you need to know if an AI can actually use those tools when it hears speech.

Lu: It’s about creating a way to take those existing text benchmarks and convert them into controlled audio evaluations without you having to re-annotate everything. They call this dataset-agnostic framework because it works with any text benchmark, which is pretty ambitious.

Meng: Ambitious is one word for it. From what I’ve seen, the challenge isn't just converting the data; it's making sure that conversion actually tests something meaningful about the models versus just moving a text task to an audio input.

Lalam: The summary of this paper is that they use a systematic TTS pipeline, combining three different TTS models—Gemini-two point five-Flash-TTS3, Gemini-two point five-Pro-TTS4, and GPT-4o-Mini-TTS—with speaker variation and environmental noise from the DEMAND dataset to make paired text and audio instances.

Tom: That’s how they generate these pairs while keeping the original tool schemas and gold labels intact so you don't have to rebuild the whole benchmark structure. They are testing seven omni-modal models on audio-converted versions of Confetti and When2Call.

Jane: It sounds like they’re trying to isolate whether a model fails because it misunderstood the text or because it misunderstood the sound of your voice.

Lu: That’s the core idea, Jane. They evaluate each task in two aligned conditions: one with just text, and one with direct audio input. This lets them estimate how much performance you lose when you switch from text to speech while keeping everything else fixed.

Meng: But what does that actually tell us about choosing between a cascade pipeline and an end-to-end omni-modal model? Is it a definitive way to pick one over the other?

Title and authors: Tom: The results show that performance is strongly dependent on both the model and the specific task, not just which architecture you pick. They found that Gemini-three point one-Flash-Live gets the highest Confetti score of seventy point four, while GPT-Realtime-one point five performs best on When2Call with a score of seventy-one point nine <ref:2605.15104#pg1,Gemini-3.1-Flash-Live>.

Jane: That’s interesting because it suggests you shouldn't just assume one architecture is universally better across all tasks and all models. It points toward picking something specific to the job at hand, doesn't it?

Lu: Right. And they showed that when they look closely at where things break down in voice failures, the errors most often reflect misunderstandings of argument values when speech is involved.

Meng: Argument value errors are a key finding for production. It means even if the model hears you correctly, it might still mess up the specific parameters you asked for in the tool call.

Tom: They found that this argument-value error category is the largest for Gemini-three point one-Flash-Live at fifty-seven point two percent and GPT-Realtime-one point five at fifty-four point three percent. That’s a lot of errors stemming from misinterpreting what you actually wanted to do with the tool input.

Jane: So, it’s not just about the audio quality; it’s about how well the model maps your spoken request to the correct parameters for that tool. That makes sense for a customer support scenario.

Lu: And they set up a diagnostic protocol by comparing three settings: clean text, direct voice input, and then the ASR-to-text cascade pipeline. This lets them see how much performance you lose when moving from text to speech, and it shows that the cost of moving from text to speech is model-dependent rather than a fixed property.

Meng: That’s useful for deployment planning because it tells you exactly what kind of audio quality your chosen model can handle before you commit to a whole system.

Tom: They found that for Gemini-three point one-Flash-Live, the text-to-voice gap on Confetti ranges from one point eight points down to four point eight points when comparing different models <ref:2605.15104#pg3,for Gemini-3.1-Flash-Live>. That variation is significant enough to motivate model and task specific diagnostics, as they point out on page three of this paper > <ref:2605.15104#pg1>

Jane: It’s a big range, which really emphasizes that you need a tailored approach rather than just picking the highest-scoring model on one benchmark blindly.

Title and authors: Lu: And they also looked at how models handle noise. For Qwen3-Omni, its accuracy remained relatively stable across signal-to-noise ratios with only modest degradation as the noise increased.

Meng: That stability is a practical thing for me; if I’m deploying something in a busy environment like a café where the audio isn't perfect, knowing that model will handle it okay is important for reducing maintenance headaches.

Tom: And they also looked at how people evaluate these models without relying on gold annotations, using reference-free LLM-as-judge protocols. They found that open-source Qwen3 judges with at least 8B parameters exceeded eighty percent agreement with proprietary judges, which is a step toward privacy preserving evaluation.

Jane: So they’re not just measuring accuracy; they’re creating a way to check performance in a production setting without needing all those expensive gold labels for every single scenario.

Lu: The conclusion is that the audio-converted versions of trusted text benchmarks really complement purpose-built audio datasets, providing a verifiable and reproducible first-stage diagnostic that can be applied to proprietary enterprise tool catalogs.

Meng: So for deployment, this means you don't just pick the end-to-end omni-modal model because it sounds fancy; you have to use these diagnostics to decide if the text-to-voice gap is small enough for that architecture to work reliably.

Tom: Exactly. The framework they built gives enterprises a reusable recipe that preserves those tool schemas and gold labels, allowing them to test their proprietary data against these controlled audio evaluations effectively.

Jane: It seems like this paper provides a really solid starting point for anyone trying to move from text benchmarks into the real world of voice agent evaluation, without having to start from scratch.

Lu: And they showed that the framework’s ability to isolate failures on argument values is particularly useful for debugging complex tool interactions in a production environment.

Meng: Yeah, being able to pinpoint exactly why an AI failed on a specific parameter value instead of just failing the whole call is a huge step forward for testing agent reliability.

Lalam: I’m fairly sure this paper provides that verifiable diagnostic capability, showing how audio-converted versions of trusted text benchmarks can complement purpose-built audio datasets >

The paper's summary: Tom: So we're looking at this paper called "From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents". It’s from Md Tahmid Rahman Laskar and his team, and they’re tackling the problem of how to test these new voice agents.

Jane: Exactly. The big issue is that most tool calling benchmarks are just text-based, but real customer support happens over voice, so you need to know if an AI can actually use those tools when it hears speech.

Lu: It’s about creating a way to take those existing text benchmarks and convert them into controlled audio evaluations without you having to re-annotate everything. They call this dataset-agnostic framework because it works with any text benchmark, which is pretty ambitious.

Meng: Ambitious is one word for it. From what I’ve seen, the challenge isn't just converting the data; it's making sure that conversion actually tests something meaningful about the models versus just moving a text task to an audio input.

Lalam: The summary of this paper is that they use a systematic TTS pipeline, combining three different TTS models—Gemini-two point five-Flash-TTS3, Gemini-two point five-Pro-TTS4, and GPT-4o-Mini-TTS—with speaker variation and environmental noise from the DEMAND dataset to make paired text and audio instances.

Tom: That’s how they generate these pairs while keeping the original tool schemas and gold labels intact so you don't have to rebuild the whole benchmark structure. They are testing seven omni-modal models on audio-converted versions of Confetti and When2Call.

Jane: It sounds like they’re trying to isolate whether a model fails because it misunderstood the text or because it misunderstood the sound of your voice.

Lu: That’s the core idea, Jane. They evaluate each task in two aligned conditions: one with just text, and one with direct audio input. This lets them estimate how much performance you lose when you switch from text to speech while keeping everything else fixed.

Meng: But what does that actually tell us about choosing between a cascade pipeline and an end-to-end omni-modal model? Is it a definitive way to pick one over the other?

The paper's summary: Tom: The results show that performance is strongly dependent on both the model and the specific task, not just which architecture you pick. They found that Gemini-three point one-Flash-Live gets the highest Confetti score of seventy point four, while GPT-Realtime-one point five performs best on When2Call with a score of seventy-one point nine.

Jane: That’s interesting because it suggests you shouldn't just assume one architecture is universally better across all tasks and all models. It points toward picking something specific to the job at hand, doesn't it?

Lu: Right. And they showed that when they look closely at where things break down in voice failures, the errors most often reflect misunderstandings of argument values when speech is involved.

Meng: Argument value errors are a key finding for production. It means even if the model hears you correctly, it might still mess up the specific parameters you asked for in the tool call.

Tom: They found that this argument-value error category is the largest for Gemini-three point one-Flash-Live at fifty-seven point two percent and GPT-Realtime-one point five at fifty-four point three percent. That’s a lot of errors stemming from misinterpreting what you actually wanted to do with the tool input.

Jane: So, it’s not just about the audio quality; it’s about how well the model maps your spoken request to the correct parameters for that tool. That makes sense for a customer support scenario.

Lu: And they set up a diagnostic protocol by comparing three settings: clean text, direct voice input, and then the ASR-to-text cascade pipeline. This lets them see how much performance you lose when moving from text to speech, and it shows that the cost of moving from text to speech is model-dependent rather than a fixed property.

Meng: That’s useful for deployment planning because it tells you exactly what kind of audio quality your chosen model can handle before you commit to a whole system.

Tom: They found that for Gemini-three point one-Flash-Live, the text-to-voice gap on Confetti ranges from one point eight points down to four point eight points when comparing different models. That variation is significant enough to motivate model and task specific diagnostics, as they point out on page three of this paper >

The paper's summary: Jane: It’s a big range, which really emphasizes that you need a tailored approach rather than just picking the highest-scoring model on one benchmark blindly.

Lu: And they also looked at how models handle noise. For Qwen3-Omni, its accuracy remained relatively stable across signal-to-noise ratios with only modest degradation as the noise increased.

Meng: That stability is a practical thing for me; if I’m deploying something in a busy environment like a café where the audio isn't perfect, knowing that model will handle it okay is important for reducing maintenance headaches.

Tom: And they also looked at how people evaluate these models without relying on gold annotations, using reference-free LLM-as-judge protocols. They found that open-source Qwen3 judges with at least 8B parameters exceeded eighty percent agreement with proprietary judges, which is a step toward privacy preserving evaluation.

Jane: So they’re not just measuring accuracy; they’re creating a way to check performance in a production setting without needing all those expensive gold labels for every single scenario.

Lu: The conclusion is that the audio-converted versions of trusted text benchmarks really complement purpose-built audio datasets, providing a verifiable and reproducible first-stage diagnostic that can be applied to proprietary enterprise tool catalogs.

Meng: So for deployment, this means you don't just pick the end-to-end omni-modal model because it sounds fancy; you have to use these diagnostics to decide if the text-to-voice gap is small enough for that architecture to work reliably.

Tom: Exactly. The framework they built gives enterprises a reusable recipe that preserves those tool schemas and gold labels, allowing them to test their proprietary data against these controlled audio evaluations effectively.

Jane: It seems like this paper provides a really solid starting point for anyone trying to move from text benchmarks into the real world of voice agent evaluation, without having to start from scratch.

Lu: And they showed that the framework’s ability to isolate failures on argument values is particularly useful for debugging complex tool interactions in a production environment.

Meng: Yeah, being able to pinpoint exactly why an AI failed on a specific parameter value instead of just failing the whole call is a huge step forward for testing agent reliability.

The paper's improvements: Tom: So, after showing how they built this conversion framework, Tom and Jane are going to talk about what the authors suggest we actually do with all this data and how they plan to use these diagnostics moving forward.

Jane: That’s right. The paper isn't just about making a test; it’s about giving engineers a tool to make real architectural decisions.

Lu: They suggest that instead of just picking the model with the highest raw score, you should use this diagnostic protocol—comparing clean text versus direct voice versus the cascade pipeline—to actually decide between a cascade and an omni-modal model.

Meng: So it’s giving us a way to pick the right tool for deployment based on what breaks at scale?

Tom: Exactly. The authors say that direct omni-modal models are only attractive when that text-to-voice gap is small, while cascades remain safer when teams need vendor flexibility.

Jane: That makes sense for a production environment because you can’t just assume one system is uniformly better across all use cases and all model tiers.

Lu: They also point out that the cost of moving from text to speech isn't a fixed number; it changes depending on which specific model you pick.

Meng: And that’s important for running costs, because if the gap varies by model, you can budget for the expected performance loss better.

Tom: They also highlighted how argument-value errors are often what causes the most failures in voice scenarios, especially with models like Gemini-three point one-Flash-Live and GPT-Realtime-one point five.

Jane: So if you’re debugging a customer interaction, you know where to look—it’s likely not a poor audio signal but how the model interpreted the specific numbers or parameters of your request.

Lu: They also mentioned that using open-source judges, like Qwen3 with at least eight billion parameters, gives you ninety percent agreement with proprietary judges for reference-free evaluation.

Meng: That’s a big deal for privacy; it means you can check performance without having to expose your entire proprietary dataset to external evaluators.

Tom: The framework they built is designed to be reusable; it’s a recipe that preserves the gold labels and tool schemas, which means enterprises can apply this directly to their own proprietary text-based data.

Jane: So what does this mean for the wider field? It suggests that audio-converted versions of trusted text benchmarks are a great way to complement purpose-built audio datasets.

Lu: And they see it as a verifiable first-stage diagnostic, which helps teams build more reliable agent architectures by testing them against these controlled, reproducible conditions.

Meng: It sounds like this gives us something concrete we can use right now to start figuring out if our current omni-modal setup is actually the right move for our specific voice tasks.

Conclusion: Tom: So we’re wrapping up on this paper, "From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents". The main thing is that they developed a method to systematically convert existing text benchmarks into controlled audio evaluations using three different TTS models.

Jane: It’s about providing a way for engineers to test their voice agents reliably without having to build entirely new audio datasets from scratch.

Lu: They really emphasize the reproducibility of the framework; it preserves the original tool schemas and gold labels, which is crucial when you're trying to figure out why an AI failed.

Meng: I like that part about preserving those labels; that makes it useful for testing against proprietary data, not just some public benchmark.

Lalam: I’m confident this framework provides the first real diagnostic capability that complements purpose-built audio corpora because it lets us see the specific failure modes in paired text and audio instances.

Tom: They showed how you can compare clean text performance against direct voice input and even an ASR to text cascade pipeline to estimate exactly what you lose when switching modalities.

Jane: That comparison shows that the cost of moving from text to speech isn't a fixed property; it really depends on the specific model you’re using.

Lu: The results show that for models like Gemini-three point one-Flash-Live and GPT-Realtime-one point five, most failures in voice are related to misunderstanding argument values.

Meng: So if we're building a system, we know to focus our debugging efforts on argument parsing errors when dealing with speech input.

Tom: And they also showed that open-source judges can get eighty percent agreement with proprietary ones using Qwen3, which is a useful step for privacy in evaluation.

Jane: Overall, the paper suggests that voice agents should be selected through model and task specific evaluation rather than assuming cascade or omni-modal systems are universally better.

Lu: It gives us a reusable recipe for enterprises to test their own tool-calling data against these controlled audio evaluations effectively.

Meng: That’s solid, because it means we don't have to rely on just one benchmark when choosing an architecture; we can run the diagnostic protocol ourselves.

Tom: I think this framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio datasets very well for anyone working on agentic systems.

Jane: It seems like a really solid starting point for moving from text benchmarks into the real world of voice agent evaluation without having to start from scratch.

Lu: And they showed that the framework’s ability to isolate failures on argument values is particularly useful for debugging complex tool interactions in a production environment.

Meng: Yeah, being able to pinpoint exactly why an AI failed on a specific parameter value instead of just failing the whole call is a huge step forward for testing agent reliability.

Tom: So, that’s the gist of "From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents". It gives us a repeatable way to test voice capabilities against text-only benchmarks by creating controlled audio pairs.

Jane: It really does set a new standard for how we evaluate these multimodal systems in real-world, noisy environments.

Lu: Next time, we’ll look at something completely different, like how embedding models are improving probabilistic race prediction.