Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing
cs.AI, cs.CL, cs.LG
Submitted: 2026-04-09
Updated: 2026-10-12
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: In reinforcement learning with verifiable rewards (RLVR), challenging math questions whose sampled groups remain uniformly incorrect yield no informative update, letting easier questions dominate
Terminology
Abstract
In reinforcement learning with verifiable rewards (RLVR), challenging math questions whose sampled groups remain uniformly incorrect yield no informative update, letting easier questions dominate actor updates. This imbalance can contribute to distribution sharpening: low- k accuracy improves while problem coverage contracts. Hint scaffolding can restore informative updates by revealing a prefix of a verified teacher solution. However, existing hint-based approaches still face two issues: teacher--student distribution mismatch and the need to reduce hint exposure to match no-hint evaluation. We address these issues through two components. Distribution-Aligned Hint Synthesis (DAHS) constructs verified teacher hints conditioned on student-style responses. Backward Hint Annealing (BHA) anneals hint exposure across length buckets as bucket accuracy improves. BHA also uses per-question hint dropout to keep no-hint rollouts throughout RL training. Under the DAPO training framework, our method improves both pass@1 and pass@2048 over DAPO on AIME24/25/26 for Qwen3-1.7B-Base and Qwen3-4B-Base, and improves pass@1 on five secondary math benchmarks (Olympiad, MATH500, Minerva Math, AMC23, and GSM8K). On Llama-3.2-1B-Instruct, the gains concentrate in the high- k regime without recovering the base model's coverage. Overall, when verified teacher prefixes are available, DAHS+BHA mitigates DAPO-induced contraction of high- k problem coverage in the evaluated challenging-math RLVR settings.
Sources
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
- The Llama 3 Herd of Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Measuring Mathematical Problem Solving With the MATH Dataset
- Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
- Solving Quantitative Reasoning Problems with Language Models
- UFT: Unifying Supervised and Reinforcement Fine-Tuning
- NGRPO: Negative-enhanced Group Relative Policy Optimization
- Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
- gpt-oss-120b & gpt-oss-20b Model Card
- Learning Montezuma's Revenge from a Single Demonstration
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- HybridFlow: A Flexible and Efficient RLHF Framework
- Qwen3 Technical Report
- Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
- Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems