VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models

summary

The gist

The gist: VLA-Pro proposes a plug-and-play framework that stores task-relevant procedural memories as task-specific LoRA adapters during training and transfers these memories during inference to

In short

VLA-Pro proposes a system that stores task-specific procedural memories as LoRA adapters during training and transfers them during testing to improve cross-task generalization in Vision-Language-Action models. It enhances model performance by dynamically retrieving and fusing relevant past manipulation experiences when tackling new, unseen tasks, leading to significant success rate improvements on robotic manipulation benchmarks.

Key concepts

Procedural Memory Storage: This involves saving the procedural knowledge of a seen task into a specific LoRA adapter ($Δ heta_i$). This memory is structured as $M_i = (D_i, Δ heta_i)$, where $D_i$ is a sequence of structured states. This allows the model to store distinct manipulation skills learned from past tasks in a compact, retrievable format for later use.
Structured Procedural State: The complex procedural experience of a task is abstracted into a simple, indexable representation $z = [a, o, e, p]$. This state captures four key elements: action type ($a$), object geometric property ($o$), end-effector orientation ($e$), and target interaction point ($p$). This structured format makes the memory easier to search and match against the current task's needs.
Cross-Task Memory Retrieval: At inference time, VLA-Pro extracts a query state $ψ_t$ from the current context. It then performs retrieval over stored memories using an action-aware matching mechanism that weights different fields of the procedural states based on the current action. This ensures that only memories relevant to the specific manipulation being performed are selected.
Memory Merging and Adaptation: The top-k retrieved task memories are combined into a fused adapter ($Δ heta_{ ext{fused } t}$) using soft fusion coefficients derived from similarity scores. This fused adapter is temporarily loaded onto the base VLA model to create a specialized policy for the current action chunk, allowing the model to adapt its behavior dynamically.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Flow Matching — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models · Read on arXiv

Institute of Trustworthy Embodied AI, Fudan University

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Now, let’s talk about the title and who wrote this paper, "VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models." It’s focused on how to move that procedural experience from one task to another.

Dev: The authors are Shengyu Si, Yuanzhuo Lu, Ruimeng Yang, Ziyi Ye, Zuxuan Wu, and Yu-Gang Jiang from the Institute of Trustworthy Embodied AI at Fudan University and Shanghai Key Laboratory of Multimodal Embodied AI. They have a lot of work in this area.

Taro: So when you read that title, what it really means is they are trying to solve the problem where VLA models can't just do what humans do when they encounter a new object or scene configuration.

Rosa: Right. It’s about moving those specific manipulation skills, the procedural memory, from one seen task to a completely different unseen task using these LoRA adapters as the mechanism for that transfer.

Dev: They are proposing this plug-and-play framework where you store task-specific LoRA adapters during training so you can transfer those memories during inference when a new context comes up.

Taro: It sounds like they're trying to bridge the gap between how we think about memory—like remembering how to pick up a bottle that’s lying down—and what the current VLA models actually do in practice.

Rosa: That’s right. They are moving away from just relying on fixed architectural modules or just adding text demonstrations, and instead embedding the experience into executable parameters for retrieval.

The paper's summary: Rosa: Let’s go over what they actually say in the summary of "VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models." It lays out how they define procedural memory.

Dev: They define procedural memory for a seen task Ti as Mi equals Di, delta theta i, where Di is the sequence of structured procedural states extracted from that task, and delta theta i is the specific LoRA adapter tied to that task.

Taro: And those structured states aren't just raw pixels or text; they are abstracted into a compact representation called z which has four components: action type a, object-related geometric property o, end-effector orientation e, and target interaction point p.

Rosa: That compact representation is key because it makes the procedural experience more indexable and retrievable than just looking at raw data points. It’s about turning complex actions into something searchable.

Dev: To retrieve this at inference time, they extract a current procedural state z hat t from the multi-modal context, and then they perform retrieval over those structured states rather than just raw images or free-form language descriptions.

Taro: And the retrieval mechanism isn't just a simple match; it uses something called Task-Wise Memory Retrieval which involves an action-aware matching mechanism where the similarity between the current state z hat t and a stored memory z(m) i is computed by weighting different fields based on the current action field at t.

Rosa: So they are prioritizing retrieval based on what’s relevant to what’s happening right now, like setting a = pick object geometry wo equals zero point three five to weigh the object geometry heavily when picking an object.

The paper's improvements: Dev: Now for the specific improvements they detail in "VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models." They focus on how their design actually helps generalization.

Taro: One big improvement is the dynamic parameter fusion part. Instead of just picking one memory, VLA-Pro dynamically injects different memories into the pretrained VLA backbone for different execution stages so the policy can reuse relevant manipulation patterns while avoiding interference from unrelated task behaviors.

Rosa: That sounds like it solves a stability issue where mixing too many things up would just make the robot do something weird when it’s not supposed to. It keeps things modular but allows for task-specific adaptation during action chunks.

Dev: Then they analyze the efficacy of top-k retrieval, showing that proper memory injection and memory retrieval significantly improve cross-task generalization, which they validate on multiple simulation and real-world robotic tasks.

Taro: They report achieving up to two hundred seven percent improvement on unseen tasks when comparing VLA-Pro against backbones like π0 point 5, RDT, and X-VLA across different benchmarks <ref:2605.29562#pg3,up to 207% improvement on unseen tasks>.

Conclusion: Rosa: So wrapping up the discussion on "VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models," the main implication is that we can give general purpose policies a way to specialize better by making procedural memories retrievable and executable.

Dev: It shows that dynamic memory retrieval helps the model avoid overusing some default behavior, like always doing a "place-on-stand" action, and allows it to switch correctly to the right spatial relation memory.

Taro: But they also point out a limitation: their performance really depends on how good those procedural state extractions are and how clear the retrieval is when you have ambiguous observations or incomplete interaction history.

Rosa: And another thing they flag is that the current memory bank is limited to seen tasks, which restricts the variety of experiences available for reuse. Future work will need to build larger, continuously expandable banks from more diverse robotic experiences.

Dev: Overall, VLA-Pro demonstrates that procedural memory retrieval and adaptation provide an effective mechanism for transferring manipulation experience while preserving modularity and execution stability.

Taro: The system is effective for related cross-task transfer, but they admit it’s less capable of handling highly novel tasks that require deeper compositional reasoning or creative memory adaptation.

Rosa: So the main point is that this framework significantly enhances the specialization capability of general purpose policies and improves cross-task generalization by giving them a way to learn from experience during training and apply it dynamically at inference.

More episodes

← Home