The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much
cs.AI, cs.CL
Submitted: 2026-10-11
Updated: 2026-10-11
Comments: 50 pages, 19 figures, 7 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism
- Mapping the Increasing Use of LLMs in Scientific Papers
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
- MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
- Ragas: Automated Evaluation of Retrieval Augmented Generation
- ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
- Agent-as-a-Judge: Evaluate Agents with Agents
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Survey on Evaluation of LLM-based Agents
- Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
- Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search
- TextGrad: Automatic "Differentiation" via Text
- Large Language Models Cannot Self-Correct Reasoning Yet
- When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems