Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
cs.LG, cs.AI, stat.ML
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/YangBa78/Evaluation-the-reliability-of-LLM-as-a-Judge
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable.
Terminology
Abstract
LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.
Sources
- SCOPE: Selective Conformal Optimized Pairwise LLM Judging
- Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Measuring Massive Multitask Language Understanding
- Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- Self-Preference Bias in LLM-as-a-Judge
- Evaluating Large Language Models at Evaluating Instruction Following
- Wider and Deeper LLM Networks are Fairer LLM Evaluators
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks