VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models

The gist: VLA-Pro proposes a plug-and-play framework that stores task-relevant procedural memories as task-specific LoRA adapters during training and transfers these memories during inference to enhance cross-task generalization in Vision-Language-Action models.

VLA Models and the Generalization Challenge

Vision Language Action (VLA) models show potential for general robotic manipulation but struggle with generalizing to unseen tasks that require transferring relevant experience across objects, scenes, and action patterns In contrast to humans, who use procedural memory to leverage past experience when grasping a lying bottle, existing VLAs often fail to adapt because they rely on fixed structural priors or contextual conditioning rather than internalizing procedural experience into retrievable and executable model parameters

VLA-Pro Framework Overview

The proposed VLA-Pro framework transfers procedural memory from seen tasks to testing tasks by storing task-specific LoRA adapters as parameterized procedural memories during training At inference time, VLA-Pro retrieves relevant procedural memories based on the current multi-modal context and dynamically fuses these memories for generating the current action chunk Specifically, this involves training VLA-Pro on seen tasks to learn one task-specific LoRA adapter as procedural memory for each task The retrieved LoRA adapters are then fused into the base VLA model, enabling the policy to reuse relevant manipulation patterns while avoiding interference from unrelated task behaviors

Procedural Memory Storage and Representation

In VLA-Pro, the procedural memory of each seen task Ti is defined as Mi = (Di, ∆θi), where Di denotes the sequence of structured procedural states extracted from task Ti, and ∆θi denotes the task-specific LoRA adapter associated with that task The structured procedural state z is abstracted into a compact representation z = [a, o, e, p], where a corresponds to action type, o to object-related geometric property, e to end-effector orientation, and p to target interaction point This compact representation makes procedural experience more indexable and retrievable

Cross-Task Memory Retrieval Mechanism

To retrieve memories at inference time, VLA-Pro first extracts the current procedural state as the retrieval query zˆt = E(Ot, l, H<t), where E(·) denotes the procedural state extractor The model performs retrieval over structured procedural states rather than raw images or free-form language descriptions Task-Wise Memory Retrieval uses an action-aware matching mechanism where the similarity Sim zˆt, z(m)i is computed by weighting different fields based on the current action field aˆt, such as a = pick object geometry wo = 0.35 After computing task-level similarities for all source memories, VLA-Pro retains the top-k most relevant source memories and their similarities to proceed to memory merging

Memory Merging and Adaptation

The retrieved top-k source memories are converted into fusion coefficients using a Softmax function over the similarity scores to obtain normalized fusion weights αj for each retrieved task These coefficients are then used to merge the corresponding task-specific LoRA adapters through parameter-wise weighted summation: ∆θfused t = X j∈Kt αj∆θj VLA-Pro loads this fused adapter onto the pretrained VLA model parameters to form a temporary policy for the current action chunk, and then unloads it after execution

Experimental Validation and Results

Experiments on RoboTwin, RLBench, and real-world manipulation tasks show that VLA-Pro consistently improves cross-task generalization across multiple backbones Specifically, in real-world robotic experiments with π0.5 as the backbone, VLA-Pro increases the average success rate from 5.8% to 65.0% over six held-out tasks Furthermore, on the RLBench benchmark, VLA-Pro achieves the best average success rate over 9 zero-shot tasks by outperforming RDT, AtomicVLA, and the π0.5 baseline Ablation analyses reveal that performance improves as the retrieved memory becomes more similar to the current task

Limitations and Future Work

VLA-Pro's performance depends on the quality of procedural state extraction and memory retrieval, which can be affected by ambiguous observations or incomplete interaction history The current memory bank is limited to seen tasks, restricting the diversity of reusable procedural experiences Future work will explore building larger and continuously expandable memory banks from more diverse robotic experiences and improving the memory representation itself through hierarchical or compositional procedural states Additionally, incorporating uncertainty-aware retrieval and adaptive memory fusion could enable the model to decide when to rely on retrieved memories, when to combine multiple memories, and when to fall back to the base policy

Conclusion

VLA-Pro effectively enhances the specialization capability of general-purpose policies, improves cross-task generalization, and maintains stable execution The framework demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability This work introduces VLA-Pro, a procedural memory transfer framework for improving the cross-task generalization of Vision-Language-Action models The results collectively demonstrate that VLA-Pro effectively enhances the specialization capability of general-purpose policies, improves cross-task generalization, and maintains stable execution The main contributions include proposing VLA-Pro which enhances VLAs with retrievable and executable procedural memories, analyzing the architectural integration of parameterized memories and the efficacy of top-k retrieval, and validating the framework on multiple simulation and real-world robotic tasks achieving up to 207% improvement on unseen tasks The results show that dynamic memory retrieval helps the model avoid overusing the dominant “place-on-stand” behavior and switch to the correct spatial-relation memory This shows its ability to mitigate high-frequency execution biases The framework is effective for related cross-task transfer, but less capable of handling highly novel tasks that require deeper compositional reasoning or creative memory adaptation The system improves efficiency by caching text embeddings of source memories and reusing loaded LoRA adapters during online adaptation As a result, the extra latency introduced by VLA-Pro is relatively small compared with the total time spent on action-chunk generation and execution This demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability The results show that dynamic memory retrieval helps the model avoid overusing the dominant “place-on-stand” behavior and switch to the correct spatial-relation memory This shows its ability to mitigate high-frequency execution biases The framework is effective for related cross-task transfer, but less capable of handling highly novel tasks that require deeper compositional reasoning or creative memory adaptation The system improves efficiency by caching text embeddings of source memories and reusing loaded LoRA adapters during online adaptation As a result, the extra latency introduced by VLA-Pro is relatively small compared with the total time spent on action-chunk generation and execution This demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability The results show that dynamic memory retrieval helps the model avoid overusing the dominant “place-on-stand” behavior and switch to the correct spatial-relation memory This shows its ability to mitigate high-frequency execution biases This framework is effective for related cross-task transfer, but less capable of handling highly novel tasks that require deeper compositional reasoning or creative memory adaptation The system improves efficiency by caching text embeddings of source memories and reusing loaded LoRA adapters during online adaptation As a result, the extra latency introduced by VLA-Pro is relatively small compared with the total time spent on action-chunk generation and execution This demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability The results show that dynamic memory retrieval helps the model avoid overusing the dominant “place-on-stand” behavior and switch to the correct spatial-relation memory This shows its ability to mitigate high-frequency execution biases This framework is effective for related cross-task transfer, but less capable of handling highly novel tasks that require deeper compositional reasoning or creative memory adaptation The system improves efficiency by caching text embeddings of source memories and reusing loaded LoRA adapters during online adaptation As a result, the extra latency introduced by VLA-Pro is relatively small compared with the total time spent on action-chunk generation and execution This demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience to novel tasks while preserving modularity and execution stability The results show that dynamic memory retrieval helps the model avoid overusing the dominant “place-on-stand” behavior and switch to the correct spatial-relation memory This shows its ability to mitigate high-frequency execution biases This framework is effective for related cross-task transfer, but

1. Better cross-task generalization via dynamic parameter fusion: VLA-Pro dynamically inject[s] different memories into the pretrained VLA backbone for different execution stages, which allows the policy to reuse relevant manipulation patterns while avoiding interference from unrelated task behaviors.

2. Improved decision stability through action-aware matching: The system uses a mechanism where the action field a determines the relative weights of o, e, and p, specifically setting weights like a = pick object geometry wo = 0.35 to ensure retrieval focuses on attributes most relevant to the current action.

3. Enhanced retrieval accuracy via specialized state extraction: The model extracts a procedural state using a system prompt constrained to a JSON format, which makes the query explicit and interpretable than directly using raw visual or textual embeddings, leading to higher MRR scores against the memory bank.

4. Increased robustness in real-world manipulation: VLA-Pro improves success rates from 5.8% to 65.0% on held-out real-world tasks by mitigating high-frequency execution biases through retrieval, as shown in Figure 4b where it retrieves relevant memory to guide the correct actions.

5. Scalability of experience transfer: Future work can focus on building larger and continuously expandable memory banks from more diverse robotic experiences, enabling the model to retrieve from a broader range of manipulation patterns beyond the current limited set of seen tasks.