TPPO: Tracking-Value PPO For Language Model Reasoning
cs.LG, cs.CL
Submitted: 2026-10-12
Updated: 2026-10-12
Code: https://github.com/LINs-lab/TPPO
License: http://creativecommons.org/licenses/by/4.0/
The gist: Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning.
Abstract
Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO, a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in https://github.com/LINs-lab/TPPO.
Sources
- Evaluating Large Language Models Trained on Code
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Process Reinforcement through Implicit Rewards
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Measuring Mathematical Problem Solving With the MATH Dataset
- OpenAI o1 System Card
- VinePPO: Refining Credit Assignment in RL Training of LLMs
- SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback
- TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
- Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Qwen3 Technical Report
- What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- Start Classifying: Categorical Critics for LLM Reinforcement Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks