MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

arXiv:2606.18558 · cs.CV · Submitted 2026-06-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction".

Jane: Here's the gist of the paper. The gist The model predicts future 3D trajectories of object-attached points in a 3D world frame given a short visual history and a language description of the intended goal, and it significantly outperforms all existing motion prediction baselines on PointMotionBench Problem Formulation and Representation The task formalized is "goal-conditioned 3D point motion forecasting," which involves predicting the future 3D positions of query points over time given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal The authors argue that "object-attached 3D points in world coordinates provide a suitable representation for general motion forecasting" because this representation is "class-agnostic, view-stable, compact, and directly useful for downstream tasks" This approach contrasts with parametric 3D models which are restricted to specific categories and embodiments MolmoMotion Architecture MolmoMotion employs two complementary classes of trajectory predictors: one with an autoregressive objective and another with a flow-matching objective The input encoding involves using Molmo2 as the vision-language backbone to process RGB observation history, the action description, and point features sampled from the anchor-frame feature map The autoregressive variant encodes initial 3D query coordinates and decodes future trajectories as quantized coordinate text, where each anchor-relative coordinate is discretized into millimeter bins The flow-matching variant predicts future trajectories in continuous 3D coordinate space using a DiT decoder conditioned on Molmo2 features Data and Training Pipeline The study introduces a full stack to study the task at scale, which includes MolmoMotion-1M, a large corpus of action-described, object-grounded 3D point trajectory dataset annotated from 1.16M unconstrained videos This data is generated through an automatic annotation pipeline consisting of five stages: semantic object grounding, temporal point correspondence, metric 3D lifting, trajectory-level filtering, and video-level clipping PointMotionBench is a human-verified benchmark spanning 111 object categories and 61 motion types MolmoMotion is pretrained on the MolmoMotion-1M dataset, training two variants to support both short visual history and single-frame history settings Evaluation and Downstream Transfer MolmoMotion is evaluated on PointMotionBench using metrics like ADE (Average Displacement Error), FDE (Final Displacement Error), and PWT (Percentage Within Threshold) Experiments show that MolmoMotion "significantly outperforms all existing motion prediction methods in PointMotionBench" Furthermore, the learned 3D motion prior transfers well to downstream applications, improving training efficiency and generalization for robot manipulation It also provides effective motion guidance for generative models to synthesize videos with more realistic object motion Model Ablations and Efficiency Ablation studies show that removing the anchor-relative parameterization produces the largest single drop in ADE/FDE across all splits, indicating that "absolute coordinates is the strongest signal of design importance" The flow-matching head is roughly two orders of magnitude faster than autoregressive decoding at T=32, which is the regime that matters in closed-loop robotic control The inference cost for Flow-matching is independent of T, unlike the autoregressive variant whose cost scales linearly in N·T The model's predicted trajectories are used to condition a lightweight 3D-track-conditioned image-to-video model (DaS+MolmoMotion), which exhibits more physically plausible object motion than unconditioned baselines MolmoMotion introduces a language-guided motion predictor that forecasts future trajectories of object-attached points in a 3D world frame, and experiments show that MolmoMotion significantly outperforms prior motion prediction methods and transfers to robot manipulation and video generation The model's predicted trajectories serve as an effective control interface for downstream video generation The work concludes that MolmoMotion significantly outperforms prior motion prediction methods and transfers to robot manipulation and video generation How it works The model input is defined by N user-specified 2D query points on an object, their initial 3D positions, a short history of RGB observations, and a language description of the intended action The architecture utilizes Molmo2 as the vision-language backbone to process image tokens from RGB observation history, text tokens from the action description, and point features sampled from the anchor-frame feature map The autoregressive variant encodes both initial 3D query coordinates and future trajectories as structured text, where each anchor-relative coordinate is discretized into millimeter bins The flow-matching variant predicts future trajectories in continuous 3D coordinate space by training a DiT decoder to predict a velocity field from a noised trajectory to the clean trajectory Data Generation Details The MolmoMotion-1M dataset is constructed by aggregating seven source corpora, including EgoDex, HD-EPIC, Xperience, MolmoSpaces, DROID, YT-VIS, and Stereo4D The annotation pipeline involves semantic object grounding using LLMs to extract an object phrase and then using MolmoPoint and SAM3 to obtain an object mask Point correspondence is established by tracking query points through the video using AllTracker, followed by lifting these tracks into 3D using ViPE for metric depth estimation Trajectory filtering and smoothing are performed using an anchor-track inconsistency score and a depth-ray smoothing optimization Transfer to Robotics and Video Generation The learned 3D motion prior is shown to transfer well to robotics planning because object trajectories in 3D world space are largely embodiment-agnostic This prior improves sample efficiency, closed-loop success, and generalization for tasks like the MolmoSpaces pick-and-place task For video generation, the predicted trajectories are used to condition a 3D-track-conditioned image-to-video model (DaS+MolmoMotion), which produces videos with more physically plausible object motion The results show that MolmoMotion predicts 3D trajectories that can serve as an effective control interface for downstream video generation Limitations and Future Work MolmoMotion requires multiple forward passes to predict dense point tracks on an object, since pretraining uses only 8 query points per example due to Molmo2 context length limit in stage 2 training This sparse point setting is not sufficient to densely represent object geometry, limiting the model’s understanding of fine-grained structure and complex deformable motion Additional downstream evaluations are needed to fully establish the effectiveness of this motion pretraining task, such as closed-loop real-world robot experiments The authors note that lifting the constraint of N=8 for dense coverage at long horizons is a natural next step for representing dense object coverage at long horizons References

1: N.

Tom: Agarwal, A.

Jane: Ali, M.

Tom: Bala, Y.

Title and authors: Jane: We've talked about what MolmoMotion is, but let's quickly go over the paper itself. The title is MolmoMotion: Forecasting Point Trajectories in three dee with Language Instruction, and it was written by Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li Shuo Liu and Jiafei Duan <ref:2606.18558#pg1>.

Tom: Right. The authors are from a good mix of groups at the Allen Institute for AI and the University of Washington and UNC Chapel Hill <ref:2606.18558#pg1>. It tells you they’re pulling together different expertise to tackle this complex problem.

Lu: Looking at their setup, it shows they are really aiming for a general motion forecasting model that works across different domains, not just one narrow application <ref:2606.18558#pg2>. They are focusing on the general requirements for what kind of representation is needed for motion modeling.

Jane: So, when we look at the abstract again, it sets out their core argument: that three dee points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks <ref:2606.18558#pg1>.

Tom: That's the thesis they are arguing for why this specific way of representing things is superior to other methods <ref:2606.18558#pg2>. It’s about establishing a foundation that works universally.

Jane: So, they are not just building another prediction tool; they are proposing a fundamental way to represent motion that should be useful for many different AI tasks, which is what makes this paper interesting to me.

The paper's summary: Tom: To summarize the main technical contribution of MolmoMotion, it introduces the task of goal-conditioned three dee point motion prediction <ref:2606.18558#pg1>. This means given a short visual history, a set of three dee query points on an object of interest, and a language description of the intended goal <ref:2606.18558#pg1,given a short visual history, a set of 3D query points on>.

Jane: And the model predicts the future three dee positions of all those queried points in a metric world coordinate frame <ref:2606.18558#pg2>. This is what they are doing—predicting where every single point goes over time.

Lu: The model achieves this by training two complementary classes of motion prediction models: one predicts future trajectories autoregressively as coordinate sequences, and the other predicts future motion with flow matching as a continuous trajectory distribution <ref:2606.18558#pg3>.

Tom: That combination gives them both a discrete way and a continuous way to model the same underlying motion, which is pretty sophisticated architecture-wise.

Jane: So, they are using two different prediction strategies—autoregressive text sequences and flow matching for continuous distributions—to cover all the bases <ref:2606.18558#pg3>.

Lu: And the input encoding uses Molmo2 as the vision-language backbone to process RGB observation history, action description, and point features sampled from the anchor-frame feature map <ref:2606.18558#pg3>. That’s how they connect all those pieces together.

Tom: It’s a holistic approach that tries to capture the visual context, the text instruction, and the spatial information simultaneously.

Jane: And that whole process is designed to be general enough to handle different objects and goals across a wide range of scenarios <ref:2606.18558#pg3>.

The paper's improvements: Tom: The primary improvement they highlight is the significant performance gain on PointMotionBench, where MolmoMotion significantly outperforms all existing motion prediction methods in PointMotionBench <ref:2606.18558#pg3>.

Jane: And what’s really compelling is the transferability of that learned three dee motion prior to other applications, specifically robotics planning and video generation <ref:2606.18558#pg2>.

Lu: Because object trajectories in three dee world space are largely embodiment-agnostic <ref:2606.18558#pg3>, the motion prior learned from internet-scale human video can transfer naturally to robotics planning <ref:2606.18558#pg3>.

Tom: And that transferability translates into real gains in robotics, improving sample efficiency, closed-loop success, and generalization for robot manipulation <ref:2606.18558#pg3>.

Jane: So, this means the model learns a useful way to move objects in the physical world from just watching videos.

Lu: And for video generation, they validate that the predicted trajectories can serve as an effective control interface for downstream video generation models Improvements. The resulting videos exhibit more physically plausible object motion than unconditioned baselines, with better VBench scores like M-Smooth and Dyn-Deg Improvements.

Conclusion: Tom: So, to wrap up MolmoMotion: Forecasting Point Trajectories in three dee with Language Instruction <ref:2606.18558#pg1,MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction>. It’s a language-guided motion predictor that forecasts future trajectories of object-attached points in a three dee world frame <ref:2606.18558#pg1>.

Jane: The key takeaway is that this system significantly outperforms prior motion prediction methods and transfers well to robot manipulation and video generation Improvements. It’s a language-guided motion predictor that forecasts future trajectories of object-attached points in a three dee world frame <ref:2606.18558#pg1>.

Lu: From my side, the main thing is confirming that the learned three dee motion prior is indeed useful for robotics planning because it improves sample efficiency, closed-loop success, and generalization to unseen objects and scenes <ref:2606.18558#pg3>.

Meng: I'm just thinking about deployment constraints. We need to make sure this prior actually works reliably in a real robot environment when things aren't perfectly smooth Improvements.

Tom: We see the speed advantage of the flow-matching head, which is roughly two orders of magnitude faster than autoregressive decoding at T=thirty-two which is the regime that matters in closed-loop robotic control Improvements.

Jane: So, this paper lays out a powerful method for generating three dee motion priors that can be used for planning and creation, setting us up nicely for what's next in this research area <ref:2606.18558#pg1>.

Allen Institute for AI University of Washington UNC Chapel Hill

cs.CV

Submitted: 2026-06-17

Updated: 2026-10-19

Code: https://github.com/allenai/molmo-motion

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The model predicts future 3D trajectories of object-attached points in a 3D world frame given a short visual history and a language description of the intended goal, and it significantly

Key concepts

Object-attached 3D points in world coordinates: This is the core data representation. Instead of modeling specific objects or camera views, the model tracks points attached to an object using their absolute 3D coordinates in a fixed world frame. This makes the prediction general because it doesn't rely on knowing exactly what kind of object or where the camera is, allowing for class-agnostic forecasting.
Autoregressive variant: This prediction method encodes the initial 3D positions and future paths as structured text. The model decodes these coordinates sequentially, discretizing each coordinate into millimeter bins. This approach is good for capturing specific sequences but can be slow for long predictions.
Flow-matching variant: This method predicts continuous 3D trajectories by training a Diffusion Transformer (DiT) decoder. It learns to predict a velocity field from noisy input trajectories to the clean trajectory. This allows the model to predict smooth, continuous motion in 3D space, which is crucial for high-fidelity forecasting.

Terminology

Benchmark — 1× in this paper · explained in 2 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 3 episodes

Summary

The gist The model predicts future 3D trajectories of object-attached points in a 3D world frame given a short visual history and a language description of the intended goal, and it significantly outperforms all existing motion prediction baselines on PointMotionBench

Problem Formulation and Representation

The task formalized is goal-conditioned 3D point motion forecasting, which involves predicting the future 3D positions of query points over time given a short visual history, a set of 3D query points on an object of interest, and a language description of the intended goal The authors argue that object-attached 3D points in world coordinates provide a suitable representation for general motion forecasting because this representation is class-agnostic, view-stable, compact, and directly useful for downstream tasks This approach contrasts with parametric 3D models which are restricted to specific categories and embodiments

MolmoMotion Architecture

MolmoMotion employs two complementary classes of trajectory predictors: one with an autoregressive objective and another with a flow-matching objective The input encoding involves using Molmo2 as the vision-language backbone to process RGB observation history, the action description, and point features sampled from the anchor-frame feature map

The autoregressive variant encodes initial 3D query coordinates and decodes future trajectories as quantized coordinate text, where each anchor-relative coordinate is discretized into millimeter bins The flow-matching variant predicts future trajectories in continuous 3D coordinate space using a DiT decoder conditioned on Molmo2 features

Data and Training Pipeline

The study introduces a full stack to study the task at scale, which includes MolmoMotion-1M, a large corpus of action-described, object-grounded 3D point trajectory dataset annotated from 1.16M unconstrained videos This data is generated through an automatic annotation pipeline consisting of five stages: semantic object grounding, temporal point correspondence, metric 3D lifting, trajectory-level filtering, and video-level clipping PointMotionBench is a human-verified benchmark spanning 111 object categories and 61 motion types MolmoMotion is pretrained on the MolmoMotion-1M dataset, training two variants to support both short visual history and single-frame history settings

Evaluation and Downstream Transfer

MolmoMotion is evaluated on PointMotionBench using metrics like ADE (Average Displacement Error), FDE (Final Displacement Error), and PWT (Percentage Within Threshold) Experiments show that MolmoMotion significantly outperforms all existing motion prediction methods in PointMotionBench Furthermore, the learned 3D motion prior transfers well to downstream applications, improving training efficiency and generalization for robot manipulation It also provides effective motion guidance for generative models to synthesize videos with more realistic object motion

Model Ablations and Efficiency

Ablation studies show that removing the anchor-relative parameterization produces the largest single drop in ADE/FDE across all splits, indicating that absolute coordinates is the strongest signal of design importance The flow-matching head is roughly two orders of magnitude faster than autoregressive decoding at T=32, which is the regime that matters in closed-loop robotic control The inference cost for Flow-matching is independent of T, unlike the autoregressive variant whose cost scales linearly in N·T The model's predicted trajectories are used to condition a lightweight 3D-track-conditioned image-to-video model (DaS+MolmoMotion), which exhibits more physically plausible object motion than unconditioned baselines

Conclusion

MolmoMotion introduces a language-guided motion predictor that forecasts future trajectories of object-attached points in a 3D world frame, and experiments show that MolmoMotion significantly outperforms prior motion prediction methods and transfers to robot manipulation and video generation The model's predicted trajectories serve as an effective control interface for downstream video generation The work concludes that MolmoMotion significantly outperforms prior motion prediction methods and transfers to robot manipulation and video generation

How it works

The model input is defined by N user-specified 2D query points on an object, their initial 3D positions, a short history of RGB observations, and a language description of the intended action The architecture utilizes Molmo2 as the vision-language backbone to process image tokens from RGB observation history, text tokens from the action description, and point features sampled from the anchor-frame feature map

The autoregressive variant encodes both initial 3D query coordinates and future trajectories as structured text, where each anchor-relative coordinate is discretized into millimeter bins The flow-matching variant predicts future trajectories in continuous 3D coordinate space by training a DiT decoder to predict a velocity field from a noised trajectory to the clean trajectory

Data Generation Details

The MolmoMotion-1M dataset is constructed by aggregating seven source corpora, including EgoDex, HD-EPIC, Xperience, MolmoSpaces, DROID, YT-VIS, and Stereo4D The annotation pipeline involves semantic object grounding using LLMs to extract an object phrase and then using MolmoPoint and SAM3 to obtain an object mask Point correspondence is established by tracking query points through the video using AllTracker, followed by lifting these tracks into 3D using ViPE for metric depth estimation Trajectory filtering and smoothing are performed using an anchor-track inconsistency score and a depth-ray smoothing optimization

Transfer to Robotics and Video Generation

The learned 3D motion prior is shown to transfer well to robotics planning because object trajectories in 3D world space are largely embodiment-agnostic This prior improves sample efficiency, closed-loop success, and generalization for tasks like the MolmoSpaces pick-and-place task For video generation, the predicted trajectories are used to condition a 3D-track-conditioned image-to-video model (DaS+MolmoMotion), which produces videos with more physically plausible object motion The results show that MolmoMotion predicts 3D trajectories that can serve as an effective control interface for downstream video generation

Limitations and Future Work

MolmoMotion requires multiple forward passes to predict dense point tracks on an object, since pretraining uses only 8 query points per example due to Molmo2 context length limit in stage 2 training This sparse point setting is not sufficient to densely represent object geometry, limiting the model’s understanding of fine-grained structure and complex deformable motion Additional downstream evaluations are needed to fully establish the effectiveness of this motion pretraining task, such as closed-loop real-world robot experiments The authors note that lifting the constraint of N=8 for dense coverage at long horizons is a natural next step for representing dense object coverage at long horizons

References

[1] N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al., Cosmos world foundation model platform for physical ai

[2] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S.

Improvements for AI systems

Bold header: Improved motion representation for general tasks

This system can now predict diverse motion patterns with different language instructions by using object-attached 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. This enables the model to generalize better across different object categories and camera viewpoints compared to category-specific models.

Bold header: Enhanced robotics planning capabilities

The improved system can improve sample efficiency, closed-loop success, and generalization for robot manipulation because the learned 3D motion prior transfers well to downstream applications, leading to a successful pick-and-place task on the MolmoSpaces benchmark with a final average success rate of 76.3% compared to 19% for Molmo2 initialization.

Bold header: High-fidelity, controllable video generation

The system can generate videos with more physically plausible object motion by using predicted trajectories to guide DaS [29], a 3D point trajectory-guided image-to-video model, resulting in better VBench scores like M-Smooth and Dyn-Deg compared to unconditioned baselines.