Long-Horizon RL Needs Muon
cs.LG, cs.AI
Submitted: 2026-10-14
Updated: 2026-10-14
Comments: 7 pages, 5 figures. Work in progress
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions.
Abstract
Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.
Sources
- Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
- Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
- Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
- R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Adam: A Method for Stochastic Optimization
- Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
- Understanding R1-Zero-Like Training: A Critical Perspective
- Reward Engineering for Reinforcement Learning in Software Tasks
- Training Software Engineering Agents and Verifiers with SWE-Gym
- When Does Muon Help Agentic Reinforcement Learning?
- SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
- Proximal Policy Optimization Algorithms
- SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training
- SWE-World: Building Software Engineering Agents in Docker-Free Environments
- ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
- Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
- Single-stream Policy Optimization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks