Parameter Exploration for RLVR via Variational Learning

Parameter-space exploration, where rollouts are generated by sampling different policies from a posterior, is investigated as a complementary control lever over exploration in Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs. A family of methods called Perturbed Parameter Policy Optimization (3PO) is introduced, which uses different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, this work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.

The paper introduces three strategies for noise injection. B3PO samples a single weight perturbation per gradient step, which makes it dependent on a single weight draw and can limit group diversity. M3PO calculates advantages separately for M model perturbations before a single weight update. C3PO divides each GRPO group across N independently sampled weights and calculates advantages on the full group to maximize diversity.

The methods are validated on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks. The chunked noising approach, C3PO, achieves the best average performance across both model families. Broadly, using multiple model samples per batch performs better than using just a single sample, and rescues more zero-advantage groups than other action-space exploration baselines. Consistent improvements in model performance are found that are concentrated on harder benchmarks like AIME and LiveCodeBench.

The paper compares 3PO to three action-space exploration baselines: adding an entropy regularization term (EntReg), raising sampling temperature when entropy collapses beyond a threshold (Polaris), and applying a KL penalty to tokens with a high covariance between probability and advantage (KLCov). For all methods, the effective group size is fixed to G = 16 rollouts per batch. For M3PO, an equal-compute comparison with AdamW is reported, using M = 4 MC samples per batch to generate rollouts and compute advantages with G = 4 at a time. For C3PO, N = 4 distinct policies are sampled per batch and 4 rollouts are accumulated for each in a rollout buffer, computing advantages on 16 rollouts in total.

All 3PO variants outperform action-space exploration baselines on average. Performance gains are concentrated on the harder AIME benchmarks, outperforming GRPO by up to 4.17% on Olmo3 and 5.83% on Qwen2.5-Math. On average, using multiple model samples outperforms single-model methods. Although B3PO outperforms GRPO on individual benchmarks such as AIME-2025, AMC, and Minerva, its average performance does not improve meaningfully beyond that of the standard GRPO. M3PO uses M MC samples to reduce noise sample variance and achieves moderate performance gains over B3PO for both model families, but its effectiveness is limited due to a proportional decrease in its group size G. C3PO has the highest average performance in both model families, likely due to the increase in group diversity due to using multiple model samples.

The paper evaluates whether 3PO really explores more by pairing each method against GRPO and studying how often each method rescues zero-advantage groups. A prompt is rescued if a method produces at least one correct rollout where GRPO produced none, and lost in the opposite case. The number of degenerate and incorrect rollouts produced by each method during training as compared to GRPO is also measured. 3PO rescues more zero-advantage groups than baselines. Action-space baselines lose to GRPO on both malformed and incorrect rollout generation rates. M3PO and C3PO are the only methods to consistently produce fewer malformed and incorrect rollouts than GRPO, which suggests that parameter-space exploration does produce more useful rollouts for RLVR.

For code generation, the LLM must produce code that passes all hidden test cases. Olmo3 is trained on the CodeR1-12K dataset, which contains verified coding problems from LeetCode and TACO. All models are evaluated on LiveCodeBench-v6, which contains problems released between January 2025 and April 2025. There is a consistent gap in the reward and accuracy curves between the 3PO methods and GRPO. C3PO is the best-performing method in this regime, reaching an LCBv6 score of 15.17. M3PO narrowly outperforms B3PO and GRPO, but is probably limited by its lower group diversity. 3PO is more sample efficient and reaches GRPO's final reward within the first 50% of the training steps. 3PO methods reach a noticeably higher reward than GRPO on code generation as compared to mathematical reasoning, possibly because code generation is a harder task for the pre-RL model.

The paper also includes further analysis. The ESS (λ) is a crucial hyperparameter that controls the amount of noise added to the weights. Decreasing λ too far to 108 always hurts performance, and completely collapses training for C3PO. Increasing λ to 1010 has more nuanced results. λ = 109 is a good default, which can be raised to 1010 if training collapses due to high noise or reduced to 108 if training stagnates close to vanilla GRPO. For M3PO, the effects of variance reduction from increasing M is roughly cancelled by the reduced group diversity from a smaller G. Removing the equal-compute correction steadily and substantially outperforms the equal-compute run, as well as B3PO with G = 16. For C3PO, even modest group diversification yields a consistent benefit, as all runs with N ≥ 2 outperform the single-perturbation baseline. Scaling beyond N = 2 improves performance in earlier training steps but smaller gains in final scores. Using N = 2 − 4 perturbations is a good strategy. Initializing the IVON Hessian h from the learned SFT prior has negligible effects, suggesting that a learned distribution from SFT does not provide a clear advantage.

Improvements to AI Systems:

1. Implement C3PO (Chunked Perturbed Parameter Policy Optimization) as the default exploration strategy in RLVR pipelines. This involves dividing each GRPO group across N=2–4 independently sampled weight perturbations, computing advantages on the full group of 16 rollouts, and updating weights once per batch. This yields the highest average performance across math and code tasks, with gains up to 5.83% on AIME benchmarks.

2. Add a dynamic noise-scale scheduler for weight perturbations. Use λ=109 as the default ESS (effective sample size) for noise injection. If training collapses (reward stagnation or divergence), raise λ to 1010 to reduce noise; if training stagnates near vanilla GRPO performance, lower λ to 108 to increase exploration. This prevents catastrophic noise-induced collapse while maintaining exploration benefits.

3. Replace action-space exploration baselines (entropy regularization, temperature raising, KL covariance penalties) with parameter-space exploration. The paper shows 3PO variants consistently outperform these baselines on average, produce fewer malformed/incorrect rollouts, and rescue more zero-advantage groups—leading to more sample-efficient training (reaching GRPO’s final reward in 50% of steps).

4. For compute-constrained settings, use M3PO with M=4 Monte Carlo samples but keep group size G≥4. This reduces noise variance in advantage estimation while maintaining moderate diversity. However, if compute allows, prefer C3PO with N=2–4 perturbations, as it outperforms M3PO due to higher group diversity.

5. Integrate a zero-advantage rescue metric into training monitoring. Track how often the method produces at least one correct rollout when GRPO produces none. This allows early detection of exploration failures and enables switching between B3PO, M3PO, and C3PO based on real-time rescue rates, improving robustness on hard benchmarks like AIME and LiveCodeBench.

6. For code generation tasks, prioritize C3PO over GRPO. The paper shows a consistent gap in reward and accuracy curves favoring 3PO, with C3PO reaching LCBv6 score of 15.17. This is especially beneficial for tasks where the pre-RL model has low initial performance, as parameter-space exploration yields larger relative gains.

7. Set N=2–4 for C3PO rather than scaling beyond N=4. While larger N improves early training steps, final performance gains diminish. Using N=2–4 balances exploration diversity with computational efficiency, avoiding unnecessary overhead.

8. Skip initialization from SFT priors for the IVON Hessian. The paper finds negligible effects from using learned SFT priors, so initialize Hessian from standard defaults to simplify implementation without performance loss.

What the improved AI system can do:

• Achieve higher average downstream performance on mathematical reasoning (e.g., AIME, AMC, Minerva) and code generation (LiveCodeBench) compared to standard GRPO, with gains up to 5.83% on AIME.

• Maintain stable training with fewer degenerate rollouts (malformed or incorrect outputs), reducing wasted compute and improving sample efficiency.

• Adaptively explore in weight space to escape local optima, particularly on hard benchmarks where action-space methods fail to produce useful rollouts.

• Operate at near-identical FLOPs cost to GRPO, making it practical for large-scale LLM fine-tuning without additional compute overhead.

• Provide a robust, tunable noise schedule (λ) that prevents training collapse while maximizing exploration benefits, suitable for diverse RLVR tasks.

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.