SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
Comments: 9 pages, 2 figures, 3 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories.
Terminology
Abstract
Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
Sources
- BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
- Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
- Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
- VIMPO: Value-Implicit Policy Optimization for LLMs
- A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
- SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning
- Understanding R1-Zero-Like Training: A Critical Perspective
- Best Practice Critic Optimization
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Maximum Likelihood Reinforcement Learning
- Single-stream Policy Optimization
- Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
- VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems