Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals
cs.CL, cs.AI, cs.LG
Submitted: 2026-10-06
Updated: 2026-10-06
Comments: 15 pages, 3 figures, 10 tables. Under review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models routinely answer questions that cannot be answered from the information given, and in dialogue they answer before enough has been said.
Terminology
Abstract
Large language models routinely answer questions that cannot be answered from the information given, and in dialogue they answer before enough has been said. Unanswerability is linearly decodable from hidden states, but it is unclear which of its forms share a representation and whether the signal is useful in dialogue. We contribute a turn-labeled multi-turn benchmark (423 conversations, 1,661 labeled turn-states) and an evaluation harness with a simulated user who answers clarifying questions, and use them with six datasets and six open-weight LLMs to test how far probes for unanswerability carry. Probes transfer robustly between datasets that share a ground of unanswerability: missing information in math (AUROC 0.77-0.97) and in a passage (SQuAD 2.0<->MuSiQue, 0.77-0.90). Probes for epistemic "known-unknowns" transfer poorly to math, but this separation weakens under lexical controls and changes with layer and coordinate system, so it remains unresolved. Single-turn probes fail zero-shot to detect when a conversation becomes answerable; in-structure probes recover it, but no better than a bag-of-words classifier. A gate on the calibrated probe, with no model fine-tuning, fires on underspecified turns far more precisely than chance, and its end-task success comes within 0.08 of a gate given the true labels. Yet across four models it does not reliably beat vanilla generation or prompted consolidation. The remaining gap lies mostly in how models use a clarification, not in detection.
Sources
- The Internal State of an LLM Knows When It's Lying
- CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models
- LLMs Get Lost In Multi-Turn Conversation
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- The Hallucination Tax of Reinforcement Finetuning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering