Fast-VLA: You Only Think Once for Robot Manipulation

arXiv:2610.21389 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Fast-VLA: You Only Think Once for Robot Manipulation".

Dev: The gist The introduction introduces Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize task understanding as fixed,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Let’s talk about the paper, Fast-VLA: You Only Think Once for Robot Manipulation. The title itself suggests a very direct approach to how these vision language models interact with robot control—focusing on doing the thinking only once per episode.

Dev: The authors are Kehong Gong, Guanli Hou, Xiaoxu Zheng, Qi Wang, Mingxi Xu, Anbang Zhai, and Xiaoyu He. They’re pointing toward a specific architectural change in VLA models that they think is necessary for real-world deployment on robotics.

Taro: What does that actually mean for the field right now? Is this just a small tweak to an existing policy, or is it something more fundamental about how we structure these models?

Rosa: It feels like it’s a structural shift because they are proposing moving the heavy model execution outside of the repeated control loop. Instead of re-encoding everything with a large model every time the robot moves, they freeze that knowledge into a fixed, multi-layer key-value task memory.

Dev: So when I think about that for my work on real-time loops, it means we’re not constantly hitting the backbone execution cost for every small action; we’re only paying that cost once at the beginning to build our context.

Taro: If it's only paid once, how does this impact reliability when the world throws something unexpected at the robot during execution? Does that fixed memory become a weakness if it can’t adapt quickly?

Rosa: That’s where they tackle that directly. They show that even with this fixed memory structure, the system remains responsive because the online actor uses live observations and proprioceptive feedback to query that memory for action queries.

Dev: The system is designed so the task understanding—the context stored in that frozen VLM memory—is stable, while the control part stays dynamic enough to respond to changes in the scene. It suggests a necessary two-timescale approach.

Taro: So they're aiming for stability at the task level while maintaining responsiveness at the control level, which is what we need when dealing with unpredictable physical interactions.

Rosa: Precisely. They are separating those timescales so that you get the benefit of pre-computed knowledge without losing the ability to react to immediate sensory input during execution. The next thing we need to look at is how they build that memory itself.

The paper's summary: Dev: Moving on, let’s talk about what Fast-VLA actually does in practice, according to their summary of the paper. They explain how this architecture works by breaking it down into three main components: the frozen VLM task memory, the online actor, and the query fusion mechanism.

Rosa: Exactly. The first part is that a frozen 4B vision language model takes the instruction and those initial multi-view frames and encodes them into fixed layer-wise KV task memory just once at episode start <ref:2610.21389#pg1,a frozen 4B vision language model>. This is their core method for establishing context.

Taro: So, it’s not just storing the data; it’s structuring it in a way that allows different parts of the model to retrieve context depending on what they need later?

Dev: Right. They retain all thirty-four layers, which means the actor can retrieve task context at different VLM depths, and this fixed memory stays fixed throughout the entire episode.

Rosa: And then we have the second part: during execution, a lightweight online actor that fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These action queries are what pull context out of the fixed memory.

Taro: That sounds like they’re using learned spatial and temporal patterns to ask questions against the stored task knowledge, rather than just feeding raw pixels into a huge model again.

Dev: Spot on. The third part is that the action decoder reads visual tokens from current and historical views and then retrieves that task context from the fixed memory before the action head predicts a fifty-step bimanual action chunk in one forward pass <ref:2610.21389#pg3,the action decoder reads visual tokens from current and historical views and>.

Rosa: It’s really clever because it means the system doesn't re-run the foundation model for every small step; it just uses what it already stored to guide what happens next. It’s all about context retrieval without re-running the whole VLM.

The paper's improvements: Taro: So, beyond just saying it works, what are the concrete improvements they’ve highlighted in terms of performance and efficiency? I want to hear the numbers on how much faster things are compared to previous methods.

Dev: They show significant speedups. Fast-VLA achieves ninety-two point four six percent success on RoboTwin two point zero Clean, and under domain randomization, it hits ninety-two point four zero percent <ref:2610.21389#pg1>. But crucially, against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by ten point nine to seventeen point one times.

Rosa: That speedup is substantial for real-time systems. They also report that it only requires about thirty GiB of online resident memory, which is quite lean considering the complexity of the model they’re running during execution <ref:2610.21389#pg1>.

Taro: What about robustness? Does this fixed memory method make it more or less brittle when things get messy in the real world?

Dev: They tested that by showing that task-memory swaps keep the actor weights fixed without any additional training, but they found performance drops of thirty-five point one eight percent or twenty-nine point six six percent success when swapping the language segment, which shows it relies on episode-matched task memory.

Taro: So they’re admitting that while it’s fast, its reliance on that specific episode context is a key point to watch for deployment and safety cases.

Rosa: And they emphasize that their full thirty-four-layer instruction and views KV combination showed the highest success on both benchmarks at ninety-five point nine five/seventy-nine point one one, so more memory seems to help capture better task understanding.

Conclusion: Dev: To wrap up, the main implication of Fast-VLA is that you can establish task context once and keep the system responsive throughout execution, which works well for long-horizon tasks too. The paper shows that this architecture separates one-time task understanding from repeated closed-loop control.

Taro: For me, the most important thing is that it proves you can achieve high success rates on those longer tasks—ninety-four point zero zero percent success on short-, medium-, and long-horizon tasks respectively with Fast-VLA <ref:2610.21389#pg1>.

Rosa: It’s a solid result because it shows that this approach is an efficient way to leverage foundation models for robotics without forcing the model to re-run itself repeatedly during every small movement.

Dev: And the final point is that this architecture separates task understanding from control, and it keeps the actor weights fixed, which means you don't need extra training just to maintain performance across different episodes.

Taro: I think what’s left for us is figuring out how to make that memory more robust so it doesn't rely so strictly on that single episode context when we move from lab trials to unpredictable real-world environments.

Kehong Gong, Guanli Hou, Xiaoxu Zheng, Qi Wang, Mingxi Xu, Anbang Zhai, Xiaoyu He, Mingyuan Zhang*, §Michael Bi Mi

Huawei Technologies Co. Ltd. · Huawei Central Research Institute

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 16 pages, 6 figures, 9 tables. Project page: https://garfield-kh.github.io/FastVLA

Project page: https://garfield-kh.github.io/FastVLA

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The introduction introduces Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize task understanding as fixed, multi-layer key–value task memory and uses a lightweight

Key concepts

Frozen VLM Contextualization
A frozen 4B Vision Language Model (VLM) is used only once at the start of an episode. It processes the initial instruction and views to create a fixed, multi-layer key-value task memory. This memory acts as a static knowledge base for the entire episode, rather than being recomputed constantly.
Lightweight Online Actor
This is a small model that runs during execution. It fuses learned queries (based on time and arm position) with real-time visual and proprioceptive feedback. This actor uses the fixed task memory to generate action queries, allowing it to predict action chunks quickly without needing to re-run the massive foundation model.
Task Memory Swaps
The system tests whether the online actor relies on episode-matched task memory. When task memory is swapped, performance drops significantly. This finding confirms that the actor's success depends on having access to the specific task context established at the beginning of that particular episode.

Summary

The gist The introduction introduces Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize task understanding as fixed, multi-layer key–value task memory and uses a lightweight online actor to predict action chunks without re-running the foundation model.

How it works

  1. A frozen 4B VLM encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once at episode start The abstract states that a frozen 4B VLM encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once at episode start.

  2. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries.

  3. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model.

Task Memory Mechanism

The core idea is that task understanding changes far more slowly than the robot state, motivating a policy that establishes task context once and then stays responsive to live observations throughout execution. The frozen VLM constructs layer-wise KV task memory from the instruction and initial views at the start of an episode. Retaining all 34 layers allows the actor to retrieve task context at different VLM depths. This fixed memory remains fixed throughout the episode, and the VLM leaves the control loop after prefill.

Online Actor Functionality

The online actor continually processes current camera views and proprioception, with short visual history in the RoboTwin configuration. Query fusion combines learned queries with arm state and wrist-view CLS tokens to form action queries. The action decoder reads visual tokens from current and historical views and retrieves task context from the fixed memory. The action head predicts a 50-step bimanual action chunk in one forward pass.

Performance and Efficiency

Fast-VLA achieves 92.46% success on RoboTwin 2.0 Clean and 92.40% under domain randomization. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9–17.1× and requires 1.22 GiB of post-prefill online resident memory. The first Fast-VLA chunk takes 153.74 ms (108.87-ms VLM prefill + 44.87-ms actor).

Key Findings and Analysis

Taskmemory swaps further show that the actor relies on episode-matched task memory

The results show that task context can be established once while closed-loop control remains responsive throughout execution.

We introduce an episode-level VLA architecture that represents one-time task understanding as fixed, multi-layer memory, moving a frozen 4B VLM outside the repeated control loop

This architecture separates one-time task understanding from repeated closed-loop control.

Across RoboTwin 2.0, LIBERO, and LIBERO-Plus, we show that this separation preserves competitive closed-loop performance while reducing RoboTwin amortized inference cost by 10.9–17.1×

Fast-VLA achieves 44.87% success on RoboTwin 2.0 (Table 2).

Task-memory swaps keep the actor weights fixed, without any additional training

Test-time swaps keep the actor weights fixed, without any additional training.

Full (Ours) 34-layer instruction + views KV shows the highest success on both benchmarks (95.95/79.11)

Full memory achieves the highest success on both benchmarks (95.95/79.11).

The paper demonstrates that task context can be established once while closed-loop control remains responsive throughout execution. The actor relies on task memory matched to the current episode to guide action prediction. Task-memory swaps cause large performance drops, showing that the actor depends on episode-matched memory in our evaluation. Wrist vision is identified as a key source of closed-loop feedback. The once-per-episode construction is an efficient trade-off between success rate and VLM computation. Finally, for long-horizon tasks, Fast-VLA achieves 94.00% success on short-, medium-, and long-horizon tasks, respectively.

The gist The introduction introduces Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize task understanding as fixed, multi-layer key–value task memory and uses a lightweight online actor to predict action chunks without re-running the foundation model.

How it works

  1. A frozen 4B VLM encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once at episode start.

  2. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries.

  3. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model.

The paper demonstrates that task context can be established once while closed-loop control remains responsive throughout execution. The actor relies on task memory matched to the current episode to guide action prediction. Wrist vision is identified as a key source of closed-loop feedback. The once-per-episode construction is an efficient trade-off between success rate and VLM computation. Finally, for long-horizon tasks, Fast-VLA achieves 94.00% success on short-, medium-, and long-horizon tasks, respectively.

Improvements for AI systems

  1. Bold header: Episode-level task understanding via frozen VLM memory. This architecture represents one-time task understanding as fixed, multi-layer memory, which allows for a frozen 4B VLM encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once at episode start.

  2. Bold header: Decoupled online actor for continuous control. The system utilizes an online actor that uses live observations and robot state to query this memory and predict 50 absolute joint commands without re-running the VLM, ensuring the policy remains closed-loop even though its task memory is fixed.

  3. Bold header: Query fusion for context retrieval. The online actor employs Query fusion combines learned queries with current arm state and wrist-view CLS tokens to form action queries, which enables the system to retrieve task context from the fixed memory by combining learned spatial/temporal queries with live feedback.

  4. Bold header: High inference efficiency for manipulation tasks. Fast-VLA achieves a significant speedup, reducing RoboTwin amortized policy inference time by 10.9–17.1×, demonstrating that task context can be established once while closed-loop control remains responsive throughout execution.

  5. Bold header: Robustness to environmental perturbations via memory matching. Task-memory swaps show the actor relies on episode-matched task memory, as demonstrated by the result where swapping the language segment leads to a performance drop of 35.18% / 29.66% success.

Sources

Related papers