SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

The gist The paper proposes SCOPE-RL, a stable and quantitative entropy control framework that leverages temperature-adaptive positive samples to counteract entropy collapse in Reinforcement Learning post-training for large language models.

Problem Context

Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness. In reasoning LLMs, exploration is closely related to entropy, a measure of policy uncertainty and output diversity. Among existing methods, Group Relative Policy Optimization (GRPO) has become a widely used backbone due to its efficiency and scalability. However, GRPO suffers from a well-known failure mode: entropy collapse, where policy entropy steadily decreases, outputs become increasingly homogeneous, and exploration vanishes prematurely. These findings point to a central challenge in RL post-training for reasoning LLMs: effective improvement requires restoring exploration, and restoring exploration requires overcoming entropy collapse.

Key Findings on Entropy Dynamics

The authors identify a previously overlooked asymmetry in entropy dynamics: positive and negative samples exert opposite effects on policy entropy, and this effect reverses across temperature regimes. Specifically, high-temperature positive samples promote entropy growth, whereas negative samples suppress it. They provide a theoretical explanation for this phenomenon: when entropy decreases during policy updates, its derivative with respect to temperature is strictly positive under positive-sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it. This leads to the proposal of SCOPE-RL, a stable and quantitative entropy control framework through a regularization term constructed from temperature-adaptive positive samples.

SCOPE-RL Framework

SCOPE-RL achieves entropy control by introducing a small auxiliary policy-gradient term constructed from temperature-adjusted positive samples. The core idea is straightforward but effective: when the policy entropy H(πθold) of the previous step falls below a target threshold H0, SCOPE-RL draws a small number of high-temperature positive samples to promote exploration; when the entropy exceeds H0, it instead uses a small number of low-temperature positive samples to maintain training stability. The formal objective is defined as JSCOPE(θ) = JGRPO(θ) + α Eq∼P (Q), o∼πTθold (Oq) [1[R(q, oi) = 1] 1 oi X oi t=1 min ri,t(θ), clipri,t(θ), 1 −ϵ, 1 + ϵ.

Experimental Results and Analysis

Extensive experiments show that SCOPE-RL consistently outperforms strong RL baselines on both Pass@1 and Pass@k. On in-domain mathematical reasoning, it improves over GRPO by +3.23 on Qwen2.5-Math-7B, +2.10 on Qwen2.5-7B, and +1.96 on Qwen3-4B. Beyond Pass@1, it also surpasses both the base model and GRPO at Pass@1024 on AIME24 and AIME25. SCOPE-RL further shows that escaping entropy collapse improves performance, while the entropy–performance relation is non-monotonic and exists an optimal level of exploration in RL post-training. The quantitative analysis suggests that exploration is most beneficial when maintained within an appropriate range, with the best average performance at H0 = 0.50.

Ablation Study

Ablation studies confirm the necessity of both key components of SCOPE-RL: positive samples as regularization and temperature as a control mechanism. Removing temperature-adjusted sampling reduces the average score from 54.84 to 51.31 and leads to clear entropy collapse, showing that the auxiliary branch alone is insufficient. Replacing the positive-only auxiliary branch with mixed positive and negative samples further drops the average score to 51.56, while entropy again collapses, indicating that negative samples weaken the desired entropy signal. Using only negative samples can still regulate entropy to some extent, but its optimization performance is substantially weaker than using positive samples.

Conclusion

In this work, SCOPE-RL regulates entropy through temperature-adaptive positive-sample regularization. Experiments show that SCOPE-RL consistently improves both Pass@1 and Pass@k over RL baselines, while enabling quantitative analysis of the exploration–performance trade-off. Overall, our results suggest that maintaining a proper level of entropy is key to improving reasoning performance in RL post-training >. This work suggests that stable exploration is important for effective optimization. The paper points toward future work focusing on state-dependent exploration mechanisms and a more complete theory of entropy dynamics in token-level RL >.

--- Page 1 ---

The gist Reinforcement learning (RL) is a key paradigm for post-training large language models (LLMs), but the widely used Group Relative Policy Optimization (GRPO) often suffers from entropy collapse: exploration quickly disappears, policies converge prematurely, and sample diversity declines, ultimately harming training effectiveness.

How it works

The paper identifies a previously overlooked asymmetry in entropy dynamics: positive and negative samples exert opposite effects on policy entropy, and this effect reverses across temperature regimes. Under high-temperature sampling, positive samples increase entropy while maintaining competitive reward, whereas negative samples accelerate entropy collapse. The authors provide a theoretical explanation for this phenomenon: when entropy decreases during policy updates, the derivative of entropy with respect to temperature is strictly positive under positive-sample updates, indicating that high-temperature positive samples can counteract entropy decay, thereby slowing entropy collapse and potentially reversing it.

SCOPE-RL Mechanism

SCOPE-RL achieves entropy control by introducing a small auxiliary policy-gradient term constructed from temperature-adjusted positive samples. The core idea is straightforward but effective: when the policy entropy H(πθold) of the previous step falls below a target threshold H0, SCOPE-RL draws a small number of high-temperature positive samples to promote exploration; when the entropy exceeds H0, it instead uses a small number of low-temperature positive samples to maintain training stability. The formal objective is defined as JSCOPE(θ) = JGRPO(θ) + α Eq∼P (Q), o∼πTθold (Oq) [1[R(q, oi) = 1] 1 oi X oi t=1 min ri,t(θ), clipri,t(θ), 1 −ϵ, 1 + ϵ.

Theoretical Analysis

The authors provide a rigorous proof for the temperature-modulated entropy control mechanism by focusing on the instantaneous gradient component. Theorem 2 states that d∆Hk(T)dTT =1 > 0 if ∆Hk(T)T =1 < 0, which proves that increasing temperature systematically introduces a relative entropy-increasing component to the update when collapse is occurring.

Limitations

The theoretical analysis is developed under several simplifying assumptions, including Assumption 3 regarding orthogonal gradients, which decouples the complex interactions between tokens and allows for tractability. The authors acknowledge that Theorem 1 focuses on the regime where entropy decreases during vanilla RL post-training and entropy collapse emerges. Empirical results consistently show that SCOPE-RL achieves effective and efficient entropy control in realistic training settings, even with a very small auxiliary budget.

Conclusion

In this work, SCOPE-RL regulates entropy through temperature-adaptive positive-sample regularization. Experiments show that SCOPE-RL consistently improves both Pass@1 and Pass@k over RL baselines, while enabling quantitative analysis of the exploration–performance trade-off. Overall, our results suggest that maintaining a proper level of entropy is key to improving reasoning performance in RL post-training >.

--- Page 2 ---

The gist The paper proposes SCOPE-RL, a stable and quantitative entropy control framework that leverages temperature-adaptive positive samples to counteract entropy collapse in Reinforcement Learning post-training for large language models.

1. Implement SCOPE-RL in reasoning LLMs post-training to prevent entropy collapse by using a regularization term constructed from temperature-adaptive positive samples. This mechanism ensures that high-temperature positive samples promote entropy growth, whereas negative samples suppress it, thereby stabilizing exploration and maintaining diversity during training.

2. Develop a quantitative analysis of the exploration performance trade-off by monitoring the entropy level throughout RL post-training. This allows researchers to identify the optimal degree of exploration for RL post-training in reasoning LLMs, as shown by results indicating that exploration is most beneficial when maintained within an appropriate range.

3. Create a system capable of dynamically adjusting its sampling strategy based on current policy uncertainty. Specifically, if policy entropy falls below a target threshold H0, the system should draw high-temperature positive samples to promote exploration; conversely, if entropy exceeds H0, it should use low-temperature positive samples to maintain training stability.

4. Enhance mathematical reasoning capabilities by applying SCOPE-RL to models like Qwen2.5-Math-7B and Qwen3-4B, which shows improvements over GRPO on benchmarks like AIME24 and MATH. The improved system can achieve performance gains such as +3.23 on Qwen2.5-Math-7B and surpass baselines at Pass@1024 for these reasoning tasks.

5. Improve out-of-domain generalization by utilizing SCOPE-RL, which transfers well beyond in-domain mathematical training data to knowledge benchmarks like ARC Challenge and MMLUPro. This suggests the system can effectively improve reasoning capabilities in scenarios outside the initial training distribution, achieving leading performance on these tests.