Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Inference
cs.LG, cs.CV, cs.RO
Submitted: 2026-07-06
Updated: 2026-10-10
Comments: 19 pages, 3 figures, 8 tables. Project page: https://corl-team.github.io/qantara
Project page: https://corl-team.github.io/qantara
License: http://creativecommons.org/licenses/by/4.0/
The gist: Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels.
Terminology
Abstract
Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels. Published JEPA world models for pixel-based control (PLDM, DINO-WM, LeWM, V-JEPA-2-AC) each commit at training time to a single inference paradigm, trajectory optimisation in a learned dynamics model, so a checkpoint trained to plan cannot be read off as a policy without retraining. Deferring that choice to inference would let the constraints that bind at deployment, per-step latency and whether a goal observation can be supplied, pick the path. We present Qantara, an end-to-end JEPA of about 21M parameters whose joint training objective pairs a Brownian-bridge interpolant between consecutive clean latents on the state axis with noise-to-data flow matching on the action axis. One checkpoint serves three inference paradigms without retraining: latent planning, behaviour-cloning action sampling, and a video-inverse composition that predicts the next latent without action conditioning, then recovers the action bridging the two. Training samples the two noise levels along the edges of the (action-noise, state-noise) square, one edge per inference dispatch, so each path is trained where it is queried. Replacing that sampler with uniform sampling over the square at matched compute drops Push-T planning from 90.1 to 53.3 success rate (SR). On the LeWM control suite at a matched 10-epoch budget, Qantara averages 91.2 SR over the four environments and reaches 93.7 on OGBench-Cube, +7.7 over DINO-WM and +17.0 over LeWM, each measured against the stronger of that baseline's published score and our retrain of it. From the same weights the two goal-blind paths, which receive no goal image, reach 82.1-83.2 SR on Push-T and 70.8-72.8 on Cube at roughly 15-65x lower inference cost, and fall to near-noise on Reacher, whose released training corpus comes from a random policy.
Sources
- Is Conditional Generative Modeling all you need for Decision-Making?
- Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
- RynnVLA-002: A Unified Vision-Language-Action and World Model
- WorldVLA: Towards Autoregressive Action World Model
- Value-guided action planning with JEPA world models
- World Models for Learning Dexterous Hand-Object Interactions from Human Videos
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
- Training Agents Inside of Scalable World Models
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- Planning with Diffusion for Flexible Behavior Synthesis
- I$^2$SB: Image-to-Image Schr\"odinger Bridge
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
- Causal-JEPA: Learning World Models through Object-Level Latent Masking
- OGBench: Benchmarking Offline Goal-Conditioned RL
- DeepMind Control Suite
- What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
- Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks