AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
cs.LG, cs.CL, cs.CR
Submitted: 2025-12-19
Updated: 2026-10-09
Comments: Accepted into NeurIPS 2026 main track
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F gap on one hidden state.
Terminology
Abstract
LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F gap on one hidden state. We show this readout is shallow enough that short, low-perplexity tokens flip the verdict from "No" to "Yes". These tokens are sampled from the judge's own next-token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches > 90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-prompt flip records will be released under responsible disclosure.
Sources
- A General Theoretical Paradigm to Understand Learning from Human Preferences
- Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models
- Adversarial Training of Reward Models
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
- ODIN: Disentangled Reward Mitigates Hacking in RLHF
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Building Guardrails for Large Language Models
- The Llama 3 Herd of Models
- Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
- Gemma: Open Models Based on Gemini Research and Technology
- Measuring Mathematical Problem Solving With the MATH Dataset
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Debiasing Reward Models by Representation Learning with Guarantees
- Language Models are Injective and Hence Invertible
- Training language models to follow instructions with human feedback
- Discovering Language Model Behaviors with Model-Written Evaluations
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks