Value as Memory: Online Value-Guided Post-training for General Robot Policies

The gist The Value as Memory (VaM) framework addresses challenges in real-world post-training for generalist robot policies by using value caching and online reinforcement learning to mitigate partial observability and improve sample efficiency.

Value Prediction

The system learns a distributional value function to estimate task progress, where the scalar value is defined as the expected categorical return over a fixed support of atoms The TD target moves each mass pψ¯(zk st+H) to Rt:t+H + γH(1 − dt)zk before linear projection onto Z The scalar advantage estimate Abt = Rt:t+H + γH(1 − dt)Vψ(st+H) − Vψ(st) is then used for policy improvement.

Value Caching and Soft Aliasing Analysis

Value caching conditions actions on recent value predictions as task-progress memory, which helps distinguish states requiring different actions The analysis formalizes how augmenting the policy input with the value cache can relax a representation-induced lower bound on policy prediction error by defining the difference between action distributions as εp. This suggests that because επ ≥ εobs, the value cache can weaken the aliasing lower bound by making two states easier to separate.

Online RL Framework and Training Stages

VaM utilizes a hybrid training data scheme that combines offline demonstrations, off-policy trajectories, and current on-policy rollouts for sample-efficient joint adaptation of policy and value models The training regime progresses from offline RL warmup through human-assisted collection to autonomous learning with sparse rewards. The four stages are:

1. Offline demonstration initialization, which trains the value expert with the progress objective Linit and the action expert with the behavior-cloning objective LBC.

2. Offline RL warmup, where trajectories are temporally downsampled to traverse them more times within a fixed update budget.

3. Online DAgger, which co-adapts both experts to the current deployment distribution while human corrections continue to supervise recovery behavior.

4. Learn from Experience (LfE), which continues online improvement using sparse terminal rewards from autonomous rollouts without collecting new human corrections.

Performance Gains and Ablations

Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of 29.4% with value caching while achieving a 32.8% gain with online autonomous learning over matched offline training. The evaluated online schedule performs 8× as many policy iterations under the same gradient-step budget, with more than 2× lower recorded time per gradient step with concurrent data collection and training.

Qualitative Results

Value caching supports progress-aware recovery, showing more progress-ordered PCA trajectories and recovery from stuck states in qualitative comparisons on Laundry Folding and Magnetic Bus Assembly. Furthermore, under partial observability, the policy using original cached values predicts opening the right gripper wider compared to using demo-trained or post-warmup values as the cache.

Sample Efficiency Enhancement

VaM exceeds Offline RL on all three tasks by 32.8% on average after LfE, supporting online closed-loop policy updates under the evaluated budgets VaM reduces mean experimental time per gradient step by 57.8% across tasks by running data collection, value training, and action training concurrently.

Limitations

The framework has limitations, including the dependence of value caching on a value function that closely tracks actual task progress, necessitating offline RL warmup for stable online training Additionally, the current real-world RL framework is limited to a single robot.

The paper presents VaM, a framework that uses recent value predictions as task-progress memory and combines offline demonstrations, off-policy trajectories, and on-policy rollouts for online policy and value adaptation. Value caching addresses action ambiguity under partial observability, while the offline-to-online RL pipeline supports continued learning from deployment experience. Together, these components offer a practical approach to post-training generalist robot policies.

--- Page 1 ---

VALUE AS MEMORY: ONLINE VALUE-GUIDED POSTTRAINING FOR GENERAL ROBOT POLICIES

Wenye Yu1,2, Yijin Chen1, Kaiyi Xiong1, Wenbo Han1 Wendi Chen1,2 Yang Jin1 Han Xue1 Jun Lv1,3 Cewu Lu1,2,3† Chuan Wen†

ABSTRACT

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of 29.4% with value caching, while achieving a 32.8% gain with online autonomous learning over matched offline training. The evaluated online schedule performs 8× as many policy iterations under the same gradient-step budget, with more than 2× lower recorded time per gradient step with concurrent data collection and training.

Value Prediction

Policy w/o Value Caching

Policy w/ Value Caching

Partial Observability Reduced Ambiguity

Offline RL methods time-consuming & delayed feedback! Online Real-world RL Checkpoint publish Data Upload fast iterations & timely feedback! 8× 2× gradient steps Long-horizon Deformable Contact-rich +33% +30%

Figure 1: VaM uses online real-world RL and value caching to boost sample efficiency and mitigate partial observability in long-horizon, deformable-object, and contact-rich manipulation.

--- Page 2 ---

1 INTRODUCTION

Real-world reinforcement learning (RL) enables generalist robot policies to learn from physical interaction and improve on deployment tasks (Chen et al., 2025b; Physical Intelligence et al., 2025). Initialized from offline expert demonstrations, a policy aims to maximize cumulative task rewards (Luo et al., 2024; Frans et al., 2025) and optionally learns from human-intervention data during deployment (Luo et al., 2025a; Wang et al. 2026). However, current real-world RL methods face limitations in data reuse and policy-update frequency. On-policy updates primarily use fresh rollouts, limiting the reuse of historical experience (Chen et al., 2025a). Iterated offline RL separates data collection from optimization, delaying feedback.

--- Page 3 ---

from newly encountered states (Physical Intelligence et al., 2025). Besides, partial observability poses another challenge. With limited observation histories, visually similar states may require different actions. Meanwhile, value functions provide compact task-progress estimates, but typically serve as filtering scores or optimization weights (Physical Intelligence et al., 2025; Frans et al., 2025).

1. No Value Caching to mitigate partial observability: The system can distinguish states requiring different actions by using cached progress estimates, as the cache indicates whether estimated progress improves, stalls, or declines. This allows the policy to select appropriate actions even when visual observations are similar but task progress differs.

2. Hybrid Training Data Scheme for Sample Efficiency: The improved system can achieve sample-efficient joint adaptation of policy and value models by combining offline demonstrations, off-policy trajectories, and current on-policy rollouts. This allows the model to learn rapidly from diverse data sources across the offline-to-online pipeline.

3. Online Autonomous Learning with Sparse Rewards: The system can transition from offline RL warmup to online autonomous learning with sparse rewards, enabling it to perform timely policy improvement through online real-world reinforcement learning. This allows the robot to learn complex, long-horizon tasks effectively without relying solely on dense reward signals.

4. Policy Iteration Frequency: The framework can perform 8× as many policy iterations under the same gradient-step budget, leading to more rapid adaptation because it achieves more than 2× lower recorded time per gradient step with concurrent data collection and training. This drastically improves the speed of closed-loop learning.

5. Soft Aliasing Mitigation: By conditioning policies on value caches, the system addresses soft aliasing, where visually similar states require different high-advantage action distributions, ensuring that the policy does not struggle to match target action distributions when inputs are ambiguous.