Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges
cs.CL
Submitted: 2026-10-13
Updated: 2026-10-13
License: http://creativecommons.org/licenses/by/4.0/
The gist: Rubric-based reward models (RMs) evaluate responses against explicit criteria.
Abstract
Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.
Sources
- Concrete Problems in AI Safety
- R3: Robust Rubric-Agnostic Reward Models
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- JudgeLRM: Large Reasoning Models as a Judge
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
- TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation
- UltraFeedback: Boosting Language Models with Scaled AI Feedback
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
- PAL: Program-aided Language Models
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Distilling the Knowledge in a Neural Network
- RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
- RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
- Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- PaperBench: Evaluating AI's Ability to Replicate AI Research
- Checklists Are Better Than Reward Models For Aligning Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering