ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation
cs.RO, cs.LG
Submitted: 2026-05-09
Updated: 2026-09-26
Code: https://github.com/MINT-SJTU/Evo-RL
Project page: https://procvlm.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful.
Terminology
Abstract
Long-horizon robotic manipulation requires dense feedback that reflects how a task advances through its procedural stages, not merely whether the final outcome is successful. Existing reward models often rely on trajectory-level success labels or time-based interpolation, which can conflate elapsed time with true task progress and therefore fail to capture unfinished steps, stagnation, and failure states. We present ProcVLM, a progress-aware vision-language model that learns procedure-grounded progress as a dense reward signal for manipulation. Rather than deriving progress from terminal outcomes or temporal proxies, ProcVLM grounds progress estimation in procedural structure and intra-stage visual change, and further adopts a reasoning-before-estimation paradigm that infers the remaining atomic actions before estimating task progress. Specifically, we construct this supervision by synthesizing frame-level subtask-semantic annotations, assigning progress budgets according to subtask structure, and distributing each budget based on intra-subtask visual change. To train ProcVLM at scale, we build a standardized procedural supervision synthesis pipeline and construct ProcCorpus-60M from 30 embodied datasets with 60M annotated frames, from which we derive ProcVQA for procedure-aware pretraining, with progress estimation as the central task alongside action segmentation and future planning. Experiments on ProcVQA and reward-model benchmarks show that ProcVLM improves embodied procedural reasoning and yields more discriminative trajectory-internal progress estimates than representative baselines, supporting its use as a dense reward model for downstream reward-guided policy optimization. Project page: https://procvlm.github.io/
Sources
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- Qwen3-VL Technical Report
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- RT-1: Robotics Transformer for Real-World Control at Scale
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- PaLM-E: An Embodied Multimodal Language Model
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
- SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
- CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning
- RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OpenVLA: An Open-Source Vision-Language-Action Model
- Reinforcement Learning with Action Chunking
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- Vision Language Models are In-Context Value Learners
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- Octo: An Open-Source Generalist Robot Policy
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- ARC: A Reasoning Recipe for Robot Foundation Models
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies