Localize-Then-Decide Guarantees for LLM Judgments
cs.CL
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026. Code: https://github.com/llm2409/Localize-Then-Decide
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains
Terminology
Abstract
Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces confidence-thresholding methods that provide such guarantees for pairwise comparisons, relying on the assumption that higher estimated confidence implies lower disagreement risk with humans. However, this assumption can break down when the number of candidate responses increases, since distributing probability mass across many alternatives can distort confidence estimates. To address this issue, we propose a Localize-Then-Decide framework. First, conformal prediction localizes a small shortlist that contains the human-preferred response with high probability. Then, a calibrated confidence-based rule selectively chooses a single response from this shortlist or abstains. This design restores the monotonic relationship between confidence and disagreement risk and enables high-probability agreement guarantees. Experiments with multiple candidate sizes across several datasets and judge LLMs demonstrate that our framework consistently achieves higher guarantee success rates and substantially higher coverage than single-stage baselines.
Sources
- Conformal Risk Control
- SCOPE: Selective Conformal Optimized Pairwise LLM Judging
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Training Verifiers to Solve Math Word Problems
- Multicalibration for Confidence Scoring in LLMs
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- Language Models (Mostly) Know What They Know
- Conformal Prediction with Large Language Models for Multi-Choice Question Answering
- Generative Judge for Evaluating Alignment
- POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization
- Split and Merge: Aligning Position Biases in LLM-based Evaluators
- Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
- Language Models with Conformal Factuality Guarantees
- Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
- Conformal Language Modeling
- Large Language Models are Inconsistent and Biased Evaluators
- JudgeBench: A Benchmark for Evaluating LLM-based Judges
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering