Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training
cs.LG, cs.AI, cs.CL
Submitted: 2026-10-07
Updated: 2026-10-07
Comments: 39 pages (9 main, 30 appendix), 12 figures (5 main, 7 appendix)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solution over tables or databases.
Terminology
Abstract
Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solution over tables or databases. Existing evaluation and training lack a leakage-aware foundation. Task success mixes the agent's disambiguation and solution-generation capabilities and can also reflect oracle leakage, that is, information that a user simulator reveals beyond what a real user would. Existing datasets also lack a shared representation of ambiguities and access boundaries. We introduce the notion of an ambiguous verifiable task, which formalizes ambiguities and resolutions, decomposing the agent into an asking policy and a solution policy, and the environment into an oracle and verifier. This framework provides baselines and metrics for evaluating task disambiguation separately from solution generation, formal definitions of oracle leakage, judge-free leakage diagnostics, and a training objective for the asking policy. We instantiate the framework in text-to-SQL with AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation specifying what the agent, oracle, and verifier may access. We evaluate clarification strategies and oracle leakage, and train an asking policy with reinforcement learning. The trained asker improves our disambiguation metrics on all six datasets and task success on five, and our leakage diagnostics measure how training affects oracle leakage.
Sources
- CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification
- Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
- Reasoning-Table: Exploring Reinforcement Learning for Table Reasoning
- Interactive Text-to-SQL via Expected Information Gain for Disambiguation
- Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction
- HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
- Qwen3 Technical Report
- One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
- $\pi$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
- SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks