ISO: An RLVR-Native Optimization Stack
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "ISO: An RLVR-Native Optimization Stack".
Jane: The gist Reinforcement learning with verifiable rewards (RLVR) can reuse the base model’s weight spectra while acquiring new behavior by changing the associated input and output singular frames,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So, wrapping up the "ISO: An RLVR-Native Optimization Stack" paper by Hanqing Zhu, Wenyan Cong, Zhizhou Sha, Sagnik Mukherjee, Xinyuan Song, David González-Martínez-Pan1, Xiaoxia Wu4 and Yuandong Tian5 <ref:2607.19331#pg1>. It’s basically proposing this ISO framework to be the way RLVR methods should be built <ref:2607.19331#pg20>.
Jane: And the main point is that spectral inheritance is a recurring structure in RLVR, and it means we can reuse the source model’s weight spectra while acquiring new behavior through changes in the associated input and output singular frames <ref:2607.19331#pg20>.
Lu: They take this observation about functional regularity and turn it into an explicit inductive bias for RLVR, which is what they call spectral inheritance <ref:2607.19331#pg20>.
Meng: From a deployment standpoint, the paper suggests we should design tools like ISO-Merger for offline composition and ISO-Optimizer for online learning <ref:2607.19331#pg20>.
Tom: So, the authors are suggesting that instead of just optimizing weights in weight space, we should be optimizing the frames—the input and output singular frames <ref:2607.19331#pg20>.
Jane: It’s a big idea because it connects the abstract math to what we actually see happening in model training <ref:2607.19331#pg6>.
Lu: This means that when you look at RLVR, you shouldn't just see a single process; you should see this separation between the base weights and the trainable frames <ref:2607.19331#pg20>.
Conclusion: Tom: So we're wrapping up on "ISO: An RLVR-Native Optimization Stack."
Jane: Yeah, it really boils down to this idea that we can reuse the base model’s weight spectra while getting new behavior just by tweaking the input and output singular frames.
Tom: Right, so they’ve turned this functional observation about how RLVR works into a whole optimization stack called ISO. It's not just another trick for training; it's a way to think about RLVR natively.
Jane: And they show this through two specific tools, the offline one, ISO-Merger and the online one, ISO-Optimizer. They’re basically saying you can compose models offline or train them online in a very structured way.
Tom: The results on Qwen3-4B-Base are pretty tight <ref:2607.19331#pg1>. ISO-AdamW beat AdamW at four hundred updates with one point five times fewer updates and less time, which is a big deal for efficiency when you’re actually running things.
Jane: It suggests that instead of just blindly optimizing every single weight in the huge weight space, focusing on these frames—the input and output structure—is where the real learning happens for RLVR.
Tom: That's what they're arguing. The authors are pushing this idea that you inherit the spectrum, but you have to optimize those frames instead of just everything else.
Jane: It moves the focus from a massive weight optimization problem to a more manageable frame optimization problem, which sounds like it could make RLVR much more accessible in practice.
Tom: Yeah, and that leads us right into how these results compare against other methods. We've got some interesting stuff coming up next about what this means for the broader landscape of model training.
University of Texas at Austin · UIUC · Emory University · Together AI · Recursive Superintelligence Inc. · ELLIS Institute Tübingen
cs.LG, cs.AI
Submitted: 2026-07-21
Updated: 2026-10-19
Comments: Latest version with enriched and revised results
Code: https://github.com/THUDM/slime
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: The gist Reinforcement learning with verifiable rewards (RLVR) can reuse the base model’s weight spectra while acquiring new behavior by changing the associated input and output singular frames,
Key concepts
- Spectral Inheritance
- This is the core finding that RLVR can reuse the original model's weight spectra while learning new skills. It happens because new behavior is achieved not by changing the main weights directly, but by adapting the associated input and output singular frames. Restoring these base spectra after training preserves most of the learned gains.
- Isospectral Optimization (ISO)
- ISO is a framework designed specifically for RLVR. It operates on the principle of 'inherit the spectrum, optimize the frames.' This involves parameterizing weights using adaptable frame variables (U and V) while keeping the base spectrum fixed. It provides both offline merging (ISO-Merger) and online training (ISO-Optimizer).
- Input/Output Singular Frames
- These are mathematical representations used in RLVR to define how the model interacts with its environment. Changing these frames allows the model to acquire new behaviors without altering the fundamental weight structure of the base model. Both input and output frames must remain adaptable for effective learning.
Terminology
Summary
The gist Reinforcement learning with verifiable rewards (RLVR) can reuse the base model’s weight spectra while acquiring new behavior by changing the associated input and output singular frames, leading to an RLVR-native optimization stack called Isospectral Optimization (ISO).
Spectral Inheritance
(Page 2)
The paper identifies spectral inheritance as a phenomenon where RLVR can reuse the base model’s weight spectra while acquiring new behavior by changing the associated input and output singular frames
<ref:2607.19331#pg6>. This is formalized by defining a fixed-spectrum family F (W0) where Z in R dout×din has sigma(Z) = sigma(W0) <ref:2607.19331#pg6> and showing that the base model’s weight spectra remain functionally reusable
<ref:2607.19331#pg6>. The functional reuse is tested through interventions, finding that restoring the base spectra after training preserves most acquired gains
<ref:2607.19331#pg6> and that keeping the base spectra fixed throughout training still supports strong RLVR learning
<ref:2607.19331#pg6>.
RLVR-Native Isospectral Optimization Framework (ISO)
(Page 2)
ISO is introduced as an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations
<ref:2607.19331#pg6>. It operates on the principle: inherit the spectrum, optimize the frames
<ref:2607.19331#pg6>. This is realized through a fixed-spectrum parameterization W(U, V) = UΣ0V⊤ where U and V are the adaptable frame variables <ref:2607.19331#pg6>.
Offline Expert Composition (ISO-Merger)
(Page 4)
The offline instantiation is ISO-Merger, which combines the frame changes of shared-base specialists into a single fixed-spectrum model
<ref:2607.19331#pg6>. This method recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods
<ref:2607.19331#pg6>. It involves steps like sign canonicalization, Stiefel tangent projection, and a ridge-stabilized system to target unit self-retention after retraction <ref:2607.19331#pg6>.
Online RLVR Training (ISO-Optimizer)
(Page 4)
The online instantiation is ISO-Optimizer, which applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed
<ref:2607.19331#pg6>. This process involves deriving Euclidean gradients with respect to frame variables and then applying the base optimizer independently to U and V, followed by polar retraction to restore feasibility in F(W0) <ref:2607.19331#pg6>.
Key Findings and Evaluation
(Page 4)
ISO-Optimizer improves accuracy in reported runs and reaches matched scores with substantially fewer training steps
<ref:2607.19331#pg6>. For Qwen3-8B-Base, ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 at 210 training steps
<ref:2607.19331#pg6>. The framework is shown to be more efficient than weight-space optimizers, as ISO-AdamW reaches AdamW’s end-of-run accuracy with 2.7× fewer training steps
<ref:2607.19331#pg6>.
Structural Requirements for Adaptation
(Page 8)
The analysis of transformation classes shows that both frames must remain adaptable
<ref:2607.19331#pg6>. Specifically, among the structural choices tested, a low-residual fixed-spectrum description requires both singular frames to remain adaptable
<ref:2607.19331#pg6>. This requirement recurs across sequential RL stages with distinct objectives <ref:2607.19331#pg6>.
Performance Against SFT
(Page 4)
The dimension-aware calibration demonstrates a stark contrast between RLVR and Supervised Fine-Tuning (SFT), showing that the illustrative SFT transition yields kappaspec values between 89 and 1364, two to three orders of magnitude above the dimensional reference
<ref:2607.19331#pg6>. Conversely, RLVR yields mean values between 1.4 and 3.2, remaining order-one relative to the isotropic dimensional reference
<ref:2607.19331#pg6>.
Conclusion
(Page 20)
The paper concludes that RLVR can reuse the source model’s weight spectra while acquiring new behavior through changes in the associated input and output singular frames
<ref:2607.19331#pg6>. ISO promotes this empirically and functionally supported regularity from a descriptive observation to an explicit RLVR inductive bias <ref:2607.19331#pg6>. This leads to the design of ISO-Merger for offline composition and ISO-Optimizer for online learning <ref:2607.19331#pg6>.
The paper establishes spectral inheritance as a recurring structure in RLVR: the base model’s weight spectra remain functionally reusable while new behavior is acquired through changes in the associated input and output singular frames, leading to an RLVR-native optimization stack called Isospectral Optimization (ISO).
How it works
(Page 31)
The ISO principle is parameterized by W(U, V) = UΣ0V⊤ where U and V are the adaptable frame variables <ref:2607.19331#pg6>. The offline ISO-Merger uses sign canonicalization, Stiefel tangent projection, and a ridge-stabilized system to target unit self-retention after retraction <ref:2607.19331#pg6>. The online ISO-Optimizer applies a conventional base optimizer to the frame variables under fixed base spectra, followed by polar retraction to restore feasibility in F(W0) <ref:2607.19331#pg6>.
ISO-Merger Details
(Page 42)
ISO-Merger retains each expert’s learned frames (Ui, Vi) while reusing the shared base spectrum Σ0 <ref:2607.19331#pg6>. It uses the keep ratio
to mask trailing modes and then solves a ridge-stabilized system to target unit self-retention after retraction <ref:2607.19331#pg6>.
ISO-Optimizer Details
(Page 46)
ISO-Optimizer applies the base optimizer to the singular-frame variables (U, V) under a fixed base spectrum Σ0 <ref:2607.19331#pg6>. It uses factor gradients derived from the chain rule and then applies polar retraction to restore feasibility in F(W0) <ref:2607.19331#pg6>.
Key Findings and Evaluation
(Page 44)
ISO-Optimizer improves accuracy in reported runs and reaches matched scores with substantially fewer training steps
<ref:2607.19331#pg6>. For Qwen3-8B-Base, ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 at 210 training steps
<ref:2607.19331#pg6>.
Structural Requirements for Adaptation
(Page 8)
The analysis of transformation classes shows that both frames must remain adaptable
<ref:2607.19331#pg6>. Specifically, among the structural choices tested, a low-residual fixed-spectrum description requires both singular frames to remain adaptable
<ref:2607.19331#pg6>.
Conclusion
(Page 20)
The paper concludes that RLVR can reuse the source model’s weight spectra while acquiring new behavior through changes in the associated input and output singular frames
<ref:2607.19331#pg6>. ISO promotes this empirically and functionally supported regularity from a descriptive observation to an explicit RLVR inductive bias <ref:2607.
Improvements for AI systems
-
Bold header: Spectral inheritance as an inductive bias for RLVR optimization. This allows for a
RLVR-native optimization stack
thatreuses the base spectra and exposes both singular frames as adaptable coordinates for offline expert composition and online RLVR training.
-
Bold header: ISO-Merger for data-free expert composition. This framework enables
data-free merging of shared-base RL experts
by retainingeach expert’s learned frames (Ui, Vi) while reusing the shared base spectrum Σ0,
achieving thestrongest aggregate performance among the compared data-free methods.
-
Bold header: ISO-Optimizer for efficient online RLVR training. This allows for
online learning with a chosen base optimizer such as AdamW or Muon to frame variables while keeping the base spectra fixed,
which improves accuracy and reaches matched scores insubstantially fewer training steps
(e.g.,ISO-AdamW reaches the same accuracy after only 100 training steps
). -
Bold header: Improved model composition via geometric merging. ISO-Merger uses a
geometry-aware orthogonal component
to merge experts, ensuring that the resulting model is optimized infixed-spectrum Stiefel coordinates,
which outperforms Euclidean task vector methods like Task Arithmetic by explicitly addressingsign conflicts, redundant updates, layer-wise geometry, or sparse task-specific signals.
-
Bold header: Enhanced mathematical reasoning performance. ISO-Optimizer improves accuracy on mathematical reasoning tasks by training Qwen3 models on DeepMath103K and reaching the
strongest AdamW run’s final accuracy with approximately 2.2× fewer training steps
(e.g.,ISO-AdamW reaches 0.509 after 210 training steps
). -
Bold header: Robustness against optimization hyperparameter tuning via fixed-spectrum constraints. The framework demonstrates that the
base spectra can remain fixed throughout learning,
showing that this approach yields strong gains even when comparing it to weight-space optimizers, as evidenced by Figure 4(b) wherekeeping the base spectrum fixed throughout training still permits strong gains.
Abstract
Reinforcement learning with verifiable rewards (RLVR) advances the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. We study this layer and functionally validate spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior by adapting both the associated input and output singular frames. We operationalize this finding as Isospectral Optimization (ISO), an RLVR-native optimization stack with complementary offline and online instantiations. Offline, ISO-Merger composes shared-base specialists in fixed-spectrum frame coordinates without post-merge data, rollouts, or distillation, and achieves the highest aggregate scores among the data-free merging methods. Online, ISO-Optimizer applies AdamW or Muon directly to the frame variables under fixed base spectra, improving aggregate accuracy in the reported math and coding experiments. On Qwen3-4B-Base, ISO-AdamW reaches AdamW's 400-update accuracy in 280 updates with 1.5 X fewer updates and 27% less wall-clock time, and exceeds AdamW by 2.2 points at update 400. These results support a reusable RLVR design principle: inherit the spectrum, optimize the frames.
Sources
- The Path Not Taken: RLVR Provably Learns Off the Principals
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- Maximum Likelihood Reinforcement Learning
- Muon is Scalable for LLM Training
- Fantastic Pretraining Optimizers and Where to Find Them
- RL's Razor: Why Online Reinforcement Learning Forgets Less
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- GLM-5: from Vibe Coding to Agentic Engineering
- The Invisible Leash: Why RLVR May or May Not Escape Its Origin
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- BroRL: Scaling Reinforcement Learning via Broadened Exploration
- Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
- Dion: Distributed Orthonormalized Updates
- Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
- ToolRL: Reward is All Tool Learning Needs
- MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
- Editing Models with Task Arithmetic
- Orthogonal Model Merging
- When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks