TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control

arXiv:2610.21922 · cs.RO · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control".

Dev: The gist The TS-WAM model introduces a lightweight TeacherStudent World-Action Model that addresses limitations in vision-language robot control by coupling self-supervised temporal learning with explicit future prediction to learn from…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper today, TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control, and the title itself is pretty descriptive. It’s about coupling temporal latent actions with world action.

Dev: Yeah, it sounds like they are trying to fix a problem where vision language models that control robots don't do well because they usually rely only on pre-recorded robot demonstrations. This paper introduces a way to learn from both those demos and videos of people doing stuff without needing a separate teacher model.

Taro: So, the core idea seems to be using self-supervised temporal learning from heterogeneous data, which means mixing robot actions and action-free human videos together. That’s a big move because it lets them learn more robust representations.

Rosa: Exactly. They introduce this vlm-jepa framework to learn these temporal latent action representations directly from that mix of data, using an EMA target branch that acts as a stable self-supervision mechanism without needing an external teacher model trained separately.

Dev: And then they couple that with delta-lwm to predict latent actions from the current observation and instruction, and those predicted actions are used to predict future visual features which then guide action generation at inference time.

Taro: That makes sense because it creates a causal chain: infer a latent intent, predict its visual consequence, and use that consequence to ground the actual robot action chunk. It’s trying to make the control loop more predictable.

The paper's summary: Rosa: To summarize what they did in TS-WAM, they are creating this lightweight TeacherStudent World-Action Model that handles vision language robot control by using self-supervised temporal learning and explicit future prediction. It addresses the issue of relying too much on demonstration data alone.

Dev: They build vlm-jepa to learn temporal latent actions from both robot demonstrations and action-free human videos, and they use delta-lwm to predict visual dynamics by predicting the motion target delta Ft equals Ft plus one minus Ft conditioned on the current feature and latent action.

Taro: So, they have this asymmetric pair of VLM branches, where the online X-VLM only gets the current observation and instruction to make latent X tokens that a small transformer predictor then maps to a predicted latent action. But there’s also an EMA Y-VLM which receives the current and future observations and reads the target latent action, which is used for training but kept causal for deployment.

Rosa: That's the teacher signal part, making sure they keep that temporal alignment strong during training while keeping the deployed policy focused only on what’s happening right now. They also use flow-matching action policy which consumes those predicted future features along with proprioception and an embodiment identifier to generate actions by matching a vector field of a continuous interpolation between noise and the demonstrated action chunk.

Dev: The whole training architecture is unified, optimizing temporal representation learning, visual dynamics, and robot control simultaneously through losses like Lrobot equals Lflow plus lambda TLtemp plus lambda WLworld. That means everything is trained together to ensure the temporal prediction stays aligned with the downstream control interface throughout task adaptation.

The paper's improvements: Rosa: One of the key contributions they highlight is vlm-jepa, this self-supervised framework for learning temporal latent action representations from robot demonstrations and action-free human videos. They emphasize that the EMA target encoders provide stable self-supervision for visual representation learning, which really helps because it emphasizes predictable semantic and temporal structure in the data.

Dev: The paper points out that TS-WAM transfers this principle to VLA by using a latent action extracted from a futureaware VLM as the target while the student is restricted to current observations, making sure the learned representation is directly usable for control. This coupling keeps the temporal prediction aligned with the downstream control interface throughout task adaptation.

Taro: They specifically address catastrophic forgetting by retaining ego-video supervision while learning the robot action objective, anchoring that shared temporal representation to both visual domains, which is crucial when adapting to new robot tasks.

Rosa: The paper also points out that this JEPA interface is central to converting action-free temporal supervision into useful representations for robot control and robust transfer among the ablations they tested. They found that removing the predicted feature-delta conditioning reduces LIBERO mean success by three point zero points, including a six point two-point loss on Long.

Dev: And they found that the vlm-jepa self-supervised architecture is key to using heterogeneous data effectively among their ablation studies, and removing it causes the largest degradation in performance across those tests.

Conclusion: Rosa: So, looking at the results on Franka Research three and RevoTron, TS-WAM shows strong manipulation performance, generalization, and robustness <ref:2610.21922#pg1>. The mean OOD success is sixty-two point nine two percent, which is compared to thirty-five point zero zero percent for a baseline model called pi point zero five.

Dev: They also show that the IND to OOD drop is only nine point five eight percentage points for TS-WAM, which is way better than the twenty point drop we saw for pi point zero five in those same tests. The joint latent-action and future feature pathway remains effective across different embodiments and task horizons, which shows it’s not just a fluke on one setup.

Taro: I think what this means for us is that you can learn from a diverse set of data sources—robot demos and human videos—and still get robust control performance even when the robot encounters something it hasn't seen before.

Rosa: Right. It’s about using this unified end-to-end training framework to learn from these heterogeneous robot demonstrations and action-free human videos to achieve strong manipulation performance, which is what TS-WAM delivers.

Dev: Yeah, the paper presents TS-WAM as a lightweight TeacherStudent World-Action Model for vision language robot control. It’s a model that learns from different data types together in one go, which is a neat way to tackle the complexity of vision language action tasks.

Menghao Yin, Jiaju Yin, Xiyue Dong, Lixin Xu, Zhenhui Zhang, Ke Zhang, Yating Feng, Renjing Xu

The Hong Kong University of Science and Technology (Guangzhou)

cs.RO

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: The gist The TS-WAM model introduces a lightweight TeacherStudent World-Action Model that addresses limitations in vision-language robot control by coupling self-supervised temporal learning with

Key concepts

vlm-jepa: This is a self-supervised framework that learns temporal latent-action representations from different video sources without needing an external teacher. It uses an EMA target branch to create stable supervision, focusing on learning the predictable semantic and temporal structure present in both robot demonstrations and action-free videos.
delta-lwm: This component predicts the change in visual dynamics, specifically predicting the motion target ($\Delta F_t = F_{t+1} - F_t$). This allows the model to learn a one-step visual dynamics model conditioned on current features and latent actions, enabling it to anticipate future visual states.
EMA target branch: This is a specific mechanism within vlm-jepa that uses an Exponential Moving Average (EMA) target encoder. It provides stable self-supervision for the visual representation learning process by using a slowly updated target signal derived from future observations, which helps guide the student model effectively.
Flow-matching action policy: This policy generates robot actions by matching a vector field derived from a continuous interpolation between noise and a demonstrated action chunk. It consumes current visual features, predicted future features ($ ext{F}_{t+1}$), proprioception, and an embodiment identifier to produce control signals.

Summary

The gist The TS-WAM model introduces a lightweight TeacherStudent World-Action Model that addresses limitations in vision-language robot control by coupling self-supervised temporal learning with explicit future prediction to learn from heterogeneous data without requiring separate pretraining stages.

How it works

Vlm-jepa learns temporal latent-action representations from robot demonstrations and action-free human videos using an EMA target branch without needing a separately trained external teacher.
TS-WAM couples vlm-jepa with delta-lwm to learn to predict latent actions from the current observation and instruction and uses them to predict future visual features that guide action generation at inference.
The model uses an asymmetric pair of VLM branches where the online X-VLM receives only the current observation and language instruction to produce latent-X tokens that a small transformer predictor maps to a predicted latent action.
The EMA Y -VLM receives the current and future observations and reads the corresponding target latent action, which is used to create a training-only teacher signal while keeping the deployed policy causal.
Delta-lwm predicts visual dynamics by predicting the motion target ∆Ft = Ft+1 − Ft conditioned on the current feature and latent action, allowing for a one-step visual dynamics model rather than a complete long-horizon simulator.
The flow-matching action policy consumes Ft, Fbt+1, the full VLM hidden sequence, proprioception, and an embodiment identifier to generate actions by matching the vector field of a continuous interpolation between noise and the demonstrated action chunk.
The training architecture jointly optimizes temporal representation learning, visual dynamics, and robot control through a unified end-to-end training strategy where losses like Lrobot = Lflow + λTLtemp + λW Lworld supervise the same intermediate quantities

Key Contributions

** We introduce vlm-jepa, a self-supervised framework for learning temporal latent-action representations from robot demonstrations and action-free human videos. The EMA target encoders provide stable self-supervision for visual representation learning. This objective emphasizes predictable semantic and temporal structure. TS-WAM transfers this principle to VLA by using a latent action extracted from a futureaware VLM as the target while the student is restricted to current observations, making the learned representation directly usable for control. The joint training keeps temporal prediction aligned with the downstream control interface throughout task adaptation. This joint training is intended to mitigate forgetting by retaining ego-video supervision while learning the robot action objective, anchoring the shared temporal representation to both visual domains. The JEPA interface is central to converting action-free temporal supervision into useful representations for robot control and robust transfer. This indicates that the JEPA interface is central to using the heterogeneous data effectively. The predicted feature-delta conditioning enables the action expert to use future information by feeding it into the flow head. Removing this pathway reduces LIBERO mean success by 3.0 points, including a 6.2-point loss on Long. The vlm-jepa self-supervised architecture is the key to using the heterogeneous data effectively among the LIBERO ablations, removing it causes the largest degradation. The JEPA interface is central to converting action-free temporal supervision into useful representations for robot control and robust transfer among the LIBERO ablations, removing it causes the largest degradation. The predicted feature-delta conditioning enables the action expert to use future information by feeding it into the flow head among the LIBERO ablations, removing it reduces LIBERO mean success by 3.0 points. The results on Franka Research 3 and RevoTron demonstrate that TS-WAM achieves strong manipulation performance, generalization, and robustness. The mean OOD success is 62.92%, versus 35.00% for π0.5. The IND-to-OOD drop is only 9.58 percentage points for TS-WAM, compared to 20.00 points for π0 5. The joint latent-action and futurefeature pathway remains effective across embodiments and task horizons. The paper concludes that TS-WAM learns from heterogeneous robot demonstrations and action-free human videos within a unified end-to-end training framework to achieve strong manipulation performance. The paper presents TS-WAM, a lightweight TeacherStudent World-Action Model for vision-language robot control. >TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control is the title of the paper. <ref:2610.21922#pg2> The paper presents TS-WAM, a lightweight TeacherStudent World-Action Model for vision-language robot control. >TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control is the title of the paper. <ref:2610.21922#pg2> The paper presents TS-WAM, a lightweight TeacherStudent World-Action Model for vision-language robot control. >TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control is the title of the paper. <ref:2610.21922#pg2> The paper presents TS-WAM, a lightweight TeacherStudent World-Action Model for vision-language robot control.

Improvements for AI systems

Better temporal representation learning through heterogeneous data: TS-WAM utilizes self-supervised temporal learning by mixing robot demonstrations and action-free human videos, allowing the model to learn from abundant video information without requiring a separately trained external teacher, which improves robustness to distribution shifts.
Explicit future prediction guidance for control: The system learns a causal chain where it first predicts latent intent, then uses them to predict future visual features that guide action generation at inference, ensuring the policy follows an explicit chain: infer a latent intent, predict its visual consequence, and ground that consequence into an action chunk.
Lightweight deployment with causal mapping: By removing the EMA target branch at inference, TS-WAM keeps the deployed policy lightweight, as it only uses the current observation while retaining visual anticipation learned from both data sources.
Action-relevant latent action coupling: The model ensures that future visual predictions are directly linked to control by using a mechanism where the matched code is required to explain the observed feature motion better than an unrelated code, directly testing the claimed world–action coupling.
Robustness to distribution shifts via joint training: The architecture achieves improved generalization across distribution shifts, as demonstrated by results showing TS-WAM maintains higher success in OOD conditions compared to baselines like π0.5, with the IND-to-OOD drop is only 9.58 percentage points for TS-WAM, compared to 20.00 points for π0.5.

Abstract

Many vision-language-action policies rely on action-labeled robot demonstrations and lack an explicit mechanism for predicting future visual states. These limitations hinder learning from action-free videos and can encourage overfitting to demonstration conditions, weakening generalization and robustness. We present TS-WAM, a lightweight Teacher-Student World-Action Model that addresses these challenges through self-supervised temporal learning and explicit future prediction. Its vlm-jepa framework learns from heterogeneous data, including robot demonstrations and action-free human videos, using an EMA target branch without requiring a separately trained external teacher. By coupling vlm-jepa with delta-lwm, TS-WAM learns to predict latent actions from the current observation and instruction and uses them to predict future visual features that guide action generation at inference. A unified end-to-end training architecture jointly optimizes temporal representation learning, visual dynamics, and robot control without a separate latent-action or world-model pretraining stage. The EMA target branch is removed at inference, keeping the deployed policy lightweight. Experiments on LIBERO, LIBERO-Plus, and SimplerEnv, together with real-world manipulation experiments on Franka Research 3 and RevoTron, demonstrate that TS-WAM achieves strong manipulation performance, generalization, and robustness.