DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

The gist DK-VLM introduces Dual-Kernel Attention to compute attention from both a visual token’s input position and its geometric position in the scene, using Reference-Frame Composition supervision to update reference frames across successive spatial judgments, achieving state-of-the-art performance on six spatial benchmarks.

How it works

DK-VLM computes attention from both a visual token’s input position and its geometric position in the scene using Dual-Kernel Attention, which computes attention from both a visual token’s input position and its geometric position in the scene. The semantic kernel retains the pretrained model’s native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. A learned gate fuses their outputs, meaning geometry thus enters attention as position rather than as fused reconstruction features.

Dual-Kernel Attention

The Dual-Kernel Formulation uses two relation kernels in parallel: the semantic kernel preserves the semantic, temporal, and two-dimensional image relations learned during pretraining, while the geometric kernel conditions relations among geometry-valid visual tokens on 3D points and camera poses. Each visual token i with valid geometry carries a 3D point xi in a common scene coordinate. The geometric kernel uses two forms of GTA (Miyato et al., 2024) to introduce multi-view geometry into attention: Point GTA for relations between scene points and Camera GTA for camera motion across views. Point GTA uses the processed scene points as translation anchors and the source frame’s rotation relative to the first frame. Camera GTA uses the relative camera poses, and both forms use the same GTA operation, where geometric transforms are applied to divide feature channels into point, camera, and remaining channels. The Kernel Fusion uses a separate scalar gate at each selected layer that learns the mixing ratio between the two kernels to fuse their outputs as Yli = (1 − αl) ϕlg(Sli) + αl ϕlg(Gli), where σ(gl) is the gate.

Reference-Frame Composition (RC)

Successive spatial judgments require more than just geometric input, so DK-VLM introduces Reference-Frame Composition (RC) supervision to train the model to update its reference frame across successive judgments. RC first converts each verified relation according to the direction convention of its source, then connects compatible relations along a sequence of landmarks or views to form a path C = (v0,..., vK). The heading obtained at one step provides the reference for the next, turning the path into a sequence of reference-frame updates, and the target at each step is recomputed from the source annotations. RC produces four target types: VSI angle contains exact turn sequences; VSI action contains the corresponding discrete actions for eligible chains; SPAR delta contains local yaw increments; and SPAR heading contains cumulative headings relative to the initial view.

Joint Training Objective

The two objectives are jointly optimized in a single training stage using the unified objective L = LSFT + λ(t)Lpair, where λ(t) controls the pairwise-loss weight. Pairwise Geometric Prediction decomposes the displacement into distance and unit direction to constrain its magnitude and orientation, predicting log distance yb dij and an L2-normalized direction ub ij. The pairwise loss is calculated as l(n)pair = l(n)dist + 0.5 l(n)dir, where the distance term uses a Huber loss on log distance and the direction term uses a cosine loss on direction.

Experimental Results

DK-VLM-8B achieves the highest average score across six spatial benchmarks, reaching 64.09, outperforming other methods in Table 2. RC supervision further improves multi-step spatial reasoning, yielding consistent gains on route-planning and viewpoint-related tasks. The full model preserves general video understanding on VideoMME and MVBench, with scores comparable to the Qwen SFT baseline. Ablation studies show that using Point GTA and Camera GTA together outperforms using either alone, and pairwise geometric supervision improves the model’s ability to make spatial judgments. The results demonstrate that the gains extend beyond a single benchmark, as DK-VLM-8B ranks first on four of the six benchmarks.

Conclusion

DK-VLM introduces 3D geometry as position into the attention computation of vision-language models through Dual-Kernel Attention and Reference-Frame Composition supervision, which improves spatial reasoning over multiple steps while preserving general video understanding. The full model achieves the highest average score across six spatial benchmarks among compared methods. The complementarity between geometric relation modeling and stepwise supervision is supported by the results of the ablation studies, showing that RC yields a larger gain on DK-VLM than without it.

AI Use Statement

We used AI assistants to help draft, translate, and polish the manuscript, retrieve literature, and implement code. The authors reviewed the AI-assisted content and code and take full responsibility for the accuracy and integrity of this work.

References

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo (Page 13).

Shuai Bai et al., Qwen3-VL technical report (Page 13).

Shuai Bai et al., Qwen2.5-VL technical report (Page 13).

Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh (Page 13).

Leonidas Guibas and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities (Page 13).

Danny Driess et al., PaLM-E: An embodied multimodal language model (Page 13).

Zhiwen Fan et al., VLM-3R: Visionlanguage models augmented with instruction-aligned 3d reconstruction (Page 14).

Chaoyou Fu et al., Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis (Page 14).

Xianqiang Gao et al., Q-GeoMem: Question-guided geometric memory for video spatial reasoning (Page 14).

Edward J. Hu et al., LoRA: Low-rank adaptation of large language models (Page 14).

Tianyu Huang et al., DreamPhysics: Learning physics-based 3D dynamics with video diffusion priors (Page 14).

Wenlong Huang et al., VoxPoser: Composable 3D value maps for robotic manipulation with language models (Page 14).

Dingming Li et al., ViewSpatialBench: Evaluating multi-perspective spatial localization in vision-language models (Page 15).

Haoyuan Li et al., Thinking with geometry: Active geometry integration for spatial reasoning (Page 15).

Hongxing Li et al., SpatialLadder: Progressive training for spatial reasoning in vision-language models (Page 15).

Kunchang Li et al., MVBench: A comprehensive multi-modal video understanding benchmark (Page 15).

Rui Yang et al., Cameras as relative positional encoding (Page 15).

Kun Ouyang et al., SpaceR: Reinforcing MLLMs in video spatial reasoning (Page 15).

Han Su et al., S2AM3D: Scalecontrollable part segmentation of 3D point clouds (Page 15).

Shengbang Tong et al., Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs (Page 15).

Jianyuan Wang et al., VGGT: Visual geometry grounded transformer (Page 15).

Jianyuan Wang et al., VGGT-ω (Page 15).

Peng Wang et al., Qwen2-VL:

1. Dual-Kernel Attention for Geometry as Position DK-VLM computes attention from both a visual token’s input position and its geometric position in the scene, allowing geometry to enter attention as position rather than as fused reconstruction features. This enables the model to compute relations directly from point displacements and relative view transformations, providing a more direct interface for spatial reasoning than fusing 3D reconstruction features.

2. Reference-Frame Composition for Multi-Step Reasoning The paper introduces RC supervision, which derives and supervises the reference-frame update at every step, improving multi-step spatial reasoning by providing stepwise turn and heading targets derived from verified single-step annotations. This allows the model to handle complex tasks like route planning where earlier movements change the position and heading on which later judgments depend.

3. Joint Optimization of Spatial and Semantic Objectives The system employs a unified objective, L = LSFT + λ(t)Lpair, which jointly optimizes instruction following (SFT) and pairwise geometric prediction (Lpair). This ensures that spatial reasoning is learned in parallel with semantic understanding, as the loss function includes terms like log(1 + dij) for log distance and a cosine loss on direction.

4. Geometry-Aware Attention Mechanisms The system utilizes Point GTA and Camera GTA to introduce multi-view geometry into attention, where Point GTA uses the patch’s scene point as the translation anchor, while keeping the source frame's orientation, modeling both scene point displacements and camera motion across views.

5. Robust Geometry Frontend Interchangeability DK-VLM is designed so that the geometric kernel of DK-VLM uses 3D points and camera poses aligned with visual tokens, without relying on intermediate features from a specific geometry frontend, meaning it can use different geometry frontends without retraining while maintaining spatial reasoning performance close to that of the default configuration.

6. Improved Spatial Reasoning on Complex Chains RC supervision significantly improves performance on multi-step reasoning; specifically, it improves route planning on VSI-Bench and ReVSI-32, as well as multi-step reasoning on MMSI-Bench, and helps the model apply spatial relations more consistently across reasoning steps by training on chains up to four steps.