Tail-Likelihood Reinforcement Learning
cs.LG, stat.ML
Submitted: 2026-09-02
Updated: 2026-09-09
Project page: https://zanette-labs.github.io/TailRL-website
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning typically optimizes average reward.
Terminology
Abstract
Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.
Sources
- Qwen2.5-VL Technical Report
- Kevin: Multi-Turn RL for Generating CUDA Kernels
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- Evaluating Large Language Models Trained on Code
- Self-Evolving Curriculum for LLM Reasoning
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Robust Reinforcement Learning with Distributional Risk-averse formulation
- Training Verifiers to Solve Math Word Problems
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
- What is the objective of reasoning with reinforcement learning?
- StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
- Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- KL-Regularized Reinforcement Learning is Designed to Mode Collapse
- Target Policy Optimization
- CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
- Let's Verify Step by Step
- RLTF: Reinforcement Learning from Unit Test Feedback
- The gem5 Simulator: Version 20.0+
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks