FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

The gist The FastVLA framework introduces a training-free, task-aware, and temporally adaptive visual token compression method for efficient Vision-Language-Action models.

Motivation

Vision-Language-Action (VLA) models are promising for general-purpose robot manipulation because they enable policies to predict low-level actions conditioned on natural language tasks and visual observations <ref:2610.22146/pg1, Vision-Language-Action (VLA) models have recently emerged as a compelling paradigm for general-purpose robot manipulation, enabling policies to predict low-level actions conditioned on natural language tasks and visual observations [3, 6, 14, 22, 30]> Despite their impressive capabilities, deploying these models in closed-loop control is often limited by inference efficiency <ref:2610.22146#pg2> At every control step, high-resolution (e.g., 224×224) and frequently multi-view observations are encoded into hundreds of visual tokens [10, 15] that dominate the compute and latency of the vision-language-model (VLM) backbone during action prediction, far outweighing the few text tokens of a short task <ref:2610.22146#pg2> Yet not all visual tokens contribute equally to action prediction <ref:2610.22146#pg2> In a typical manipulation step, only a small subset of tokens carries decision-critical information (e.g., the target object/place or the gripper–object contact region), while the rest are redundant or even distracting [11, 26, 27]> Reducing the number of visual tokens processed by the large language-model (LLM) backbone, without sacrificing task performance, has therefore become a key direction for accelerating VLA inference [20, 21, 25]> However, directly applying VLM-specific token compression methods [4, 7, 29] to robotic control is often suboptimal <ref:2610.22146#pg2> because these methods are designed for single-shot vision-language understanding, where the objective is to preserve global semantic content for perception tasks such as detection, captioning, or generation <ref:2610.22146#pg2> Unlike image-based VLMs that process a single image, or video-based VLMs that encode a fixed sequence of frames, a VLA interacts with the environment through actions that directly shape its subsequent observations <ref:2610.22146#pg4> at each step the agent receives a new observation conditioned on its previous action and predicts the next one <ref:2610.22146#pg4> Visual tokens are thus not representations of a static scene but components of a sequential decision process, which sets token compression for VLA apart from its VLM counterpart in two respects <ref:2610.22146#pg4> The first is that token relevance is governed by the task rather than by generic visual saliency (now standard practice in language-guided pruning), which we adopt as our importance signal rather than claim as a contribution <ref:2610.22146#pg4> The second, and the focus of this work, is that the amount of decision-critical visual information varies drastically across control steps <ref:2610.22146#pg4> some steps (e.g., target localization) rely on broad scene context, whereas others (e.g., grasping or contact) depend on a small, localized region <ref:2610.22146#pg4> A single fixed compression rate is therefore inherently mismatched, wasting computation when the useful information is concentrated and discarding necessary context when broader information is required <ref:2610.22146#pg4>

Methodology

FastVLA proposes a training-free visual token compression framework whose core is a temporally adaptive, per-step compression rate <ref:2610.22146#pg4> At each step, FastVLA scores visual-token importance with task-to-vision attention which is a training-free signal that a controlled comparison shows is more reliable for action prediction than vision-only or fused attention <ref:2610.22146#pg4> then, crucially, it decides how many tokens to keep from the shape of the resulting importance distribution <ref:2610.22146#pg4> Specifically, it estimates the effective number of tokens the decision depends on via an inverse-Simpson uniformity measure, and maps this quantity through an online sliding-window calibration to a timestep-dependent compression rate <ref:2610.22146#pg4> when the current step concentrates on few tokens relative to its recent history it prunes aggressively, and when the decision draws on broader context it prunes conservatively <ref:2610.22146#pg4>

The framework proceeds in two stages: first, it scores visual-token importance with a task-conditioned attention signal, and then, as our core contribution, sets how many tokens to keep adaptively at each step from the concentration of that importance distribution <ref:2610.22146#pg4>

1. Preliminary: Visual Token Compression in VLA Models

Methods that reduce the visual-token cost of a VLA backbone fall into three families that differ in what they do to the token sequence <ref:2610.22146#pg4> Token pruning permanently discards a subset of visual tokens at one or more layers, shortening the sequence processed by the subsequent layers; the removed tokens are not recovered <ref:2610.22146#pg4> Token merging instead aggregates similar tokens into a smaller set of representative tokens, preserving part of the discarded information through pooling rather than hard deletion <ref:2610.22146#pg4> Reuse/caching exploits temporal redundancy across control steps: token representations or key-value states computed at previous timesteps are cached and reused for regions that remain unchanged, so that only changed tokens are recomputed <ref:2610.22146#pg4> FastVLA belongs to the pruning family: at every action step it performs a hard, task-conditioned top-K selection of visual tokens, with the retained budget set adaptively per step <ref:2610.22146#pg4>

2. Task-conditioned visual token importance

Since a VLA policy is explicitly conditioned on the language task, so visual-token usage is driven by the task prompt rather than by generic visual saliency <ref:2610.22146#pg5> The tokens that matter are those covering the target object/place and the interaction region named by the task <ref:2610.22146#pg5> Since selecting tokens by attention is already standard in VLM compression, we do not propose it as new but instead re-evaluate three candidate signals for VLA control: all-to-vision (naive cross-modal attention from the fused representation to visual tokens), task-to-vision (task tokens attending to visual tokens), and vision-to-vision (visual selfattention capturing perceptual saliency) <ref:2610.22146#pg5>

Formally, let T ∈ RM×d and V ∈ R N×d denote the task- and visual-token representations <ref:2610.22146#pg5> Following scaled dot-product attention [24], we apply RMSNorm to obtain T˜ and V˜, compute affinities S = T˜ V˜ T/√ d ∈ RM×N, and apply a softmax over visual tokens to obtain attention weights A <ref:2610.22146#pg5> Averaging over a subset of task tokens Q gives a per-token importance score: si = 1 / Q X q∈Q Aq,i, i = 1... N, so that visual tokens consistently attended by the task context receive higher importance <ref:2610.22146#pg5>

3. Adaptive Step-aware Compression Rate

Having fixed which tokens matter, the central question of our method is how many to keep at each step <ref:2610.22146#pg6> In a closed-loop rollout, successive steps rely on visual information to very different extents <ref:2610.22146#pg6> For example, approaching or grasping an object depends on a small localized region, whereas locating a target or verifying the scene requires broad context <ref:2610.22146#pg6> A fixed compression rate is therefore inherently mismatched because it wastes computation when the useful information is concentrated and discards decision-critical tokens when broader context is needed, so the rate should instead adapt to this per-step concentration <ref:2610.22146#pg6> We implement this in two steps: measure how concentrated the importance distribution is, then map that concentration to a per-step rate <ref:2610.22146#pg6>

Inverse Simpson uniformity.

1. FastVLA enables closed-loop VLA deployment by setting a task-dependent compression rate based on task relevance, addressing the limitation that a single fixed compression rate is therefore inherently mismatched. This allows the system to allocate the token budget where each decision needs it, leading to reduced LLM-backbone FLOPs (35-59% in simulation) and improved action latency.

2. The system can achieve a 1.25× speedup on a physical Franka Research 3 robot for end-to-end latency, while raising the control frequency by up to 25% at comparable success, making real-time manipulation significantly more responsive than baseline methods like VLA-Cache.

3. FastVLA's importance scoring uses task-to-vision attention as a trainingfree signal that a controlled comparison shows is more reliable for action prediction than vision-only or fused attention. This ensures that tokens critical to the task, such as the target object/place, are retained, as demonstrated by its superior performance over other scoring strategies on the LIBERO benchmark.

Vision-Language-Action (VLA) models present a promising paradigm for general-purpose robot manipulation, but closed-loop deployment is often limited by inference efficiency: at each timestep, high-resolution, often multi-view images are encoded into hundreds of visual tokens that dominate transformer computation and latency. Token compression via attention is a natural solution but is challenging for VLA control because (1) decision-critical tokens are determined by the task, not generic visual saliency, and (2) the token budget varies substantially across steps, so static metrics and budgets waste computation or discard control-critical tokens. We introduce FastVLA, a training-free, task-aware, and temporally adaptive visual token compression framework. At each step, FastVLA scores token importance via task-to-vision attention and adaptively sets the compression rate from the importance-distribution uniformity, measured by an inverse-Simpson metric with online sliding-window calibration. We evaluate FastVLA on OpenVLA, OpenVLA-OFT, and CogACT across LIBERO and SIMPLER, and on a physical Franka Research 3 robot. In simulation, it reduces LLM-backbone FLOPs by 35-59% and action latency by 9-32% (a 1.09 times-1.46 times speedup) while maintaining task success, achieving a comparable or better efficiency-performance trade-off than existing plug-and-play token compression/pruning/reuse baselines. On the real robot, it cuts end-to-end latency by 20% (1.25 times speedup) at comparable success. Code is available at https://github.com/jiuyixu25/FastVLA.