Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

cs.CL

Submitted: 2026-10-13

Updated: 2026-10-13

Code: https://github.com/fabian-lu/TDM_as_a_Judge

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• Labels Override Definitions in Jev-Style Typed Decision Models
• Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
• Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
• Training Verifiers to Solve Math Word Problems
• JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications
• Evaluating and Benchmarking the System One Model Jev
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
• Judgement in the Age of Jev: From Evaluation Scarcity to Evaluation Abundance
• JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models
• Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
• JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
• The Argument and the Letterhead: Source-Position Coherence in AI Evaluation
• Benchmarking System One decision models against trained classifiers and language models for automated decision gates
• Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
• Verbosity Bias in Preference Labeling by Large Language Models
• Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It
• Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering