When Left Is Right? Camera-Motion Understanding beyond Screen
cs.CV
Submitted: 2026-10-12
Updated: 2026-10-12
Project page: https://3dagentworld.github.io/CameraVLM
License: http://creativecommons.org/licenses/by/4.0/
The gist: Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin
Abstract
Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- DepthLM: Metric Depth From Vision Language Models
- Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
- RegGS: Unposed Sparse Views Gaussian Splatting with 3DGS Registration
- Graph-Guided Scene Reconstruction from Images with 3D Gaussian Splatting
- Unposed 3DGS Reconstruction with Probabilistic Procrustes Mapping
- Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmaps
- HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction
- Local All-Pair Correspondence for Point Tracking
- ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
- RynnBrain: Open Embodied Foundation Models
- BootsTAP: Bootstrapped Training for Tracking-Any-Point
- Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
- Geometry-Guided Camera Motion Understanding in VideoLLMs
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
- Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning
- Kubric: A scalable dataset generator
- AllTracker: Efficient Dense Point Tracking at High Resolution
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models