Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

arXiv:2606.01629 · cs.CL · Submitted 2026-06-01 · Read on arXiv

cs.CL

Submitted: 2026-06-01

Updated: 2026-08-28

Comments: EMNLP 2026 main

Code: https://github.com/cjj826/LongJudgeBench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Benchmark — 39× in this paper · explained in 2 episodes
Accuracy — 30× in this paper · explained in 1 episode
LLM judge — 27× in this paper · explained in 2 episodes
LLM-as-a-Judge — 27× in this paper · explained in 3 episodes
Consistency — 10× in this paper · explained in 3 episodes
Alignment — 4× in this paper
Hallucination — 4× in this paper · explained in 4 episodes
Verification — 4× in this paper · explained in 2 episodes
Adam — 3× in this paper · explained in 2 episodes
Benchmarking — 3× in this paper · explained in 4 episodes
Emotional Resonance — 3× in this paper · explained in 2 episodes
Hallucination Detection — 3× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 3× in this paper · explained in 8 episodes
Robustness — 3× in this paper · explained in 5 episodes
Thinking mode — 3× in this paper · explained in 2 episodes
Adaptation — 2× in this paper · explained in 1 episode
Annotator Agreement — 2× in this paper
Core — 2× in this paper · explained in 2 episodes
Perplexity — 2× in this paper · explained in 3 episodes
Agentic AI — 1× in this paper · explained in 2 episodes
Calibration — 1× in this paper · explained in 8 episodes
Grounding — 1× in this paper · explained in 3 episodes
RAG — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Verifier — 1× in this paper · explained in 2 episodes

Sources

Related papers