Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
cs.RO, cs.LG
Submitted: 2026-10-07
Updated: 2026-10-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited.
Terminology
Abstract
Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited. Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning. Broader successful-mode coverage may provide alternative strategies under distribution shifts. Inspired by this, we introduce DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which turns successful-behavior diversity into an explicit RL objective. DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity. This design encourages broader coverage of feasible solutions without rewarding diverse failures or superficial timing differences. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on π 0 and 2.0 points on π 0.5. On a dual-arm AgileX PiPER-X platform, DRIVE further increases average OOD success from 64.1% to 73.3% (+9.2 points), demonstrating gains that persist under physical deployment.
Sources
- Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models
- $\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models
- Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model
- Proximal Policy Optimization Algorithms
- Interactive Post-Training for Vision-Language-Action Models
- Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models
- RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
- Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectories
- Robotic Control via Embodied Chain-of-Thought Reasoning
- Do World Action Models Generalize Better than VLAs? A Robustness Study
- Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies