Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

arXiv:2608.19515 · cs.CL · Submitted 2026-08-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does".

Jane: The gist The HEAR2ACT protocol introduces a unified evaluation protocol for text and spoken assistants to determine when prosodic evidence should guide information gathering and final selection in task-oriented dialogue,…

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we’ve walked through the main points of Hear2Act. The authors are looking at how prosodic cues impact task-oriented dialogue decisions in these four hundred eighty scenarios and they found that prosody is most useful when the words themselves don't provide enough information <ref:2608.19515#pg1>.

Tom: And what they proved is that audio-capable models don't reliably carry that prosodic information into action unless there’s a mechanism to make it explicit, like inferring the concern status from the audio and then using that text representation for the next step.

Lu: The implication is that we can't just rely on listening to speech to automatically improve decision-making in these systems; we need a specific way to bridge the gap between what we hear and what the model acts upon.

Meng: For those of us building systems, this means if we want prosody to matter, we have to design that intermediate representation carefully, because raw audio alone isn't getting us close to the performance gains they see with explicit state access.

Lalam: Also, they showed that having the correct turn-specific information changes how much interaction happens; it helps the assistant know when to keep asking questions and when it's time to stop searching for something else.

Jane: So, for someone just listening in on the show, what does this mean practically? It means that if you're talking to an AI assistant and you want it to understand your true needs—the ones you don't fully state—you need a system that can reliably decode those subtle vocal cues and turn them into something concrete for the AI to act on.

Tom: Exactly. The paper shows that we aren't seeing magic where we thought we were; it’s about building better ways to translate sound into usable context for the AI's decision engine.

Conclusion: Tom: So, we’ve seen how they set up this Hear2Act protocol to test if sound cues actually change what an assistant does in a real conversation.

Jane: It looks like the title is Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does, and that sounds like it’s really focused on finding the line between listening to speech and making a smart choice.

Tom: Yeah, it’s about these four hundred eighty scenarios where they keep the user needs fixed but change whether a hidden need is spoken or just implied by the tone of voice.

Jane: What that really means for us is that prosody—the pitch and rhythm of the voice—is carrying task-relevant information that words often leave out, and they’re showing us how to measure if it actually leads to a better outcome.

Tom: Exactly, but what I saw in the numbers was interesting because when they just fed raw audio into the model, it didn't help much at all; success rates barely moved from fourteen point three percent to fifteen point one percent.

Jane: That’s where the paper gets really important, because they found that if we make that prosodic information explicit—if the AI infers what’s missing and puts it into text—the success rate jumps up to nearly forty percent.

Tom: So, it proves the perception-action gap: raw audio is weak on its own, but making that info explicit in the text enables much better decisions.

Jane: And they also showed that even when the concern is stated right in the words, like an explicit instruction, that performance boost doesn't happen nearly as much.

Tom: It means if we want to use spoken input effectively without just relying on it being transcribed perfectly, we need a way to bridge that gap between sound and text representation.

Jane: That leads us nicely into how this kind of explicit information handling affects how the AI actually talks back and decides when to stop searching for an answer.

Amazon University of Illinois Urbana-Champaign · University College London

cs.CL

Submitted: 2026-08-20

Updated: 2026-10-18

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The HEAR2ACT protocol introduces a unified evaluation protocol for text and spoken assistants to determine when prosodic evidence should guide information gathering and final selection in

Key concepts

Prosody-mediated conditions: This condition tests whether an assistant makes better decisions based on how something is said (tone, pitch) rather than just the words spoken. The study found prosody helps when the actual need isn't stated in the text, but it doesn't always translate into better actions.
Explicit lexical conditions: This condition tests performance when all necessary information, including hidden user needs, is clearly stated in words. In this case, prosody provides little extra benefit because the verbal information is already explicit and accessible to the model.
Perception-action gap: This refers to the difference between what an AI model 'perceives' from audio (like tone) and what it actually 'acts' upon in the dialogue. The results show that while raw audio input is limited, making that audio information explicit in text allows models to make much better decisions.
Unstated user needs: These are hidden requirements or concerns a user has during a task that they do not explicitly state in their spoken request. HEAR2ACT scenarios include these unstated needs to see if prosody can reveal them, which is where the most decision-making value was found.

Terminology

Benchmarking — 1× in this paper · explained in 4 episodes

Summary

The gist The HEAR2ACT protocol introduces a unified evaluation protocol for text and spoken assistants to determine when prosodic evidence should guide information gathering and final selection in task-oriented dialogue, revealing that prosody matters when lexical evidence is insufficient but does not reliably carry it into action without an explicit intermediate representation >

How it works

HEAR2ACT evaluates whether an assistant can use task-relevant prosodic evidence to make better dialogue decisions across 480 persona-grounded scenarios, each pairing a surface request and prioritized hidden concerns with candidate options whose fit is objectively verifiable > The design keeps the task and user needs fixed while varying the assistant’s access to transcript, audio, and explicit concern-state information to isolate both the decision value of prosodic evidence and where that value is lost between audio and action >

The evaluation involves comparing two audiocapable LLMs under different feedback conditions:

Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3% >
When models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state >

This contrast largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance >

Benchmark Scenarios and Design

Each scenario combines an initial request, three prioritized hidden concerns (a hard constraint L1, a strong preference L2, and a moderate preference L3), and an 11-candidate set of options whose fit to those concerns is known > The best-fitting option satisfies all three hidden concerns but does not lead on visible attributes such as price or rating, while the other candidates remain plausible under incomplete information >

The interactive rollouts are governed by a deterministic user engine that determines which concerns remain unresolved and what information can be disclosed based on the assistant's action > The rollout ends when an option is accepted or the 20-turn budget is reached, and matched rollouts compare trajectories under different access conditions such as transcript-only versus ground-truth state access >

Key Findings on Decision Value

The results address three questions: when prosodic information has decision value, whether spoken assistants can carry that information into action, and how correct concern information changes the dialogue > Prosody adds substantial decision value when it conveys task-relevant information missing from the words, but current audio-capable LLMs make limited use of this value directly from audio >

When models infer the concern status from audio, express it in text, and use it for next-action selection, the optimal-solution rate rises to 39.6%, close to 40.7% with the ground-truth state > This contrast largely disappears under Explicit lexical feedback, where the relevant concern is already stated in words >

Impact on Dialogue Trajectory

Correct turn-specific concern information shifts interaction toward elicitation and helps the assistant determine when to continue searching and when to stop > Under Prosody-mediated feedback, the average number of ASK and CLARIFY actions rises from 1.8 to 4.4 per dialogue, indicating more information gathering before recommendation > Furthermore, state access reduces premature closure on a suboptimal option from 69.7% to 26.7% under Prosody-mediated feedback >

Conclusion

The paper concludes that prosody is most useful when lexical evidence is insufficient, but audio-capable LLMs do not reliably carry it into action without an explicit intermediate representation > Correct concern information also improves stopping behavior by helping the assistant know when to continue eliciting and searching, and when to stop > Together, these results show that audio-capable LLMs can recover useful information from speech but do not reliably carry it into action without an explicit intermediate representation >

Limitations

HEAR2ACT isolates one decision-relevant function of prosody: whether and how strongly a recommendation remains unresolved > The benchmark does not cover broader preference structures, other functions of prosody, or unconstrained first-turn recommendation > User utterances and speech are synthesized from structured states, and ground-truth-state conditions serve as diagnostic rather than deployment settings > The inferred-state condition establishes that the intervention is sufficient without isolating the mechanism >

Ethical Considerations

HEAR2ACT uses constructed scenarios and personas, with dialogue and speech generated from structured benchmark states, containing no real user conversations, identities, recordings, or personal data > The benchmark evaluates a narrow communicative signal: whether a recommendation remains unresolved > Such evidence should support clarification rather than be used to infer sensitive attributes, override explicit user statements, or make high-stakes decisions without confirmation > The human study involved volunteer graduate students who judged generated speech clips, and no personal or sensitive information was collected >

References

The paper cites numerous works including Budzianowski et al. (2018) on MultiWOZ, Si et al. (2023) on SpokenWOZ, and Deng et al./Wang et al. (2025/2026b) on HumDial-EIBench > It also references work by Chen et al. (2026) on LISTEN, Kim et al. (2026) on PALLM, and Wang et al. (2026c) on ParaBridge > The paper also cites work by Deng et al./Wang et al. (2025/2026b) on MULTIBench, and Wang et al. (2026a) on RealTalk-CN > The paper also cites work by Lin et al./Yang et al. (2024/2026) on StyleTalk and ParaS2S, and Feng et al./Lin et al. (2023/2024) on emotion-aware work > The paper also cites work by Chen et al. (2026) on Do audio LLMs really LISTEN, or just transcribe? > It also references work by Debaupte et al. (2026) on VocalAffectBench > The paper also cites work by Feng et al./Lin et al. (2024/2026) on Infusing emotions into task-oriented dialogue systems > It also references work by Kuofeng Gao et al. (2025) on Benchmarking open-ended audio dialogue understanding for large audio-language models > The paper also cites work by Wei-Ning Hsu et al. (2021) on HuBERT > It also references work by Shuzheng Si et al. (2023) on SpokenWOZ > <ref:2608.

Improvements for AI systems

Bold prosodic information integration: Implement a mechanism where audio-capable LLMs infer concern status from audio, represent it in text, and use it for next-action selection, as this raises the optimal-solution rate to 39.6% when lexical evidence is insufficient.
Explicit representation of inferred state: Design a two-stage intervention where the model first infers the task-relevant concern status from audio (Infer whether the user's prosody indicates that the current recommendation is resolved or remains unresolved) and then supplies that textual inference to the decision prompt for next-action selection.
Policy shift toward elicitation: Modify dialogue policies so that Correct concern information shifts the dialogue from early recommendation toward information gathering, specifically by increasing the average number of ASK and CLARIFY actions across the five text LLMs rises from 1.8 to 4.4 per dialogue.
Improved stopping behavior: Integrate concern-state access to improve stopping behavior, as state access reduces premature closure on a suboptimal option from 69.7% to 26.7%, allowing the assistant to continue searching when necessary rather than confirming prematurely based on insufficient evidence.