Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations

arXiv:2506.17545 · cs.CV · Submitted 2025-06-21 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning with Lightweight Annotations".

Jane: Here's the gist of the paper. The gist: Scene-R1 introduces a video-grounded framework that learns to reason about 3D scenes without any point-wise 3D instance supervision by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline.

Tom: Scene Overview and Motivation Existing 3D-aware LLMs act as black boxes because they output bounding boxes or textual answers without revealing how those decisions are made, and they still rely on pre-trained 3D detectors to supply object proposals<ref:2506.17545#pg2>.

Jane: Scene-R1 addresses this by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline<ref:2506.17545#pg3>.

Tom: This approach eliminates the need for 3D detector-based proposals while capturing fine geometry and material cues through tracking objects using SAM2 to produce pixel-accurate masks in RGB frames, which are then projected back into 3D<ref:2506.17545#pg3>.

Paper summary: Tom: So we've looked at the core idea of the paper "Scene-R1: Video-Grounded Large Language Models for three dee Scene Reasoning with Lightweight Annotations" and how they use reinforcement learning to drive a two stage grounding pipeline for three dee scene reasoning <ref:2506.17545#pg1,Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning>. The central thesis is that existing black box three dee LLMs need to be replaced because they rely on pre trained three dee detectors and don't show their work <ref:2506.17545#pg1,rely on pre trained 3D detectors>.

Jane: They claim Scene-R1 avoids those reliance by pairing reinforcement learning driven reasoning with a two stage grounding pipeline, which means the model learns to reason about the video first, then ground it in image space with a 2D bounding box before finally refining that into a precise three dee localization <ref:2506.17545#pg1,by pairing reinforcement learning driven reasoning with a two stage grounding pipeline>.

Lu: The paper claims they achieve this without using any point wise three dee instance supervision by instead analyzing the video to select relevant snippets for temporal grounding and then predicting a 2D bounding box on each frame <ref:2506.17545#pg1,any point wise 3D instance supervision by>. After that, they use SAM2 to produce pixel accurate masks in RGB frames which are then projected back into three dee space <ref:2506.17545#pg1,SAM2 to produce pixel accurate>.

Meng: They also mention that their training pipeline is efficient because it only needs task level 2D boxes or textual labels without dense three dee point wise labels, avoiding the need to train models for dense three dee instance segmentation models <ref:2506.17545#pg1>. That’s a big relief for anyone thinking about shipping these things in production.

Lalam: And they show that they surpass previous best label free baselines on multiple datasets, and they also provide transparent intermediate reasoning steps through chain of thought rationales as part of the results. I'm fairly sure this is what makes the method stand out to people who care about how AI makes decisions.

Tom: So we've established that Scene-R1 is a video grounded framework that learns to reason about three dee scenes without point wise three dee instance supervision by pairing reinforcement learning driven reasoning with a two stage grounding pipeline <ref:2506.17545#pg1,point wise 3D instance supervision by pairing reinforcement learning driven reasoning with>. This sets the stage for what’s next in multimodal understanding.

Conclusion: Jane: So looking at the paper "Scene-R1: Video-Grounded Large Language Models for three dee Scene Reasoning with Lightweight Annotations" and its authors, it really shows that we can get much better performance in this area without needing to collect massive amounts of expensive three dee annotations <ref:2506.17545#pg1,Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning>.

Tom: I agree. The implication is that video centric perception combined with reinforcement learning based reasoning offers a practical route to transparent and scalable three dee scene understanding that doesn't require the cost of dense three dee annotation <ref:2506.17545#pg1>.

Lu: It suggests that RGB-D video paired with reinforcement learning post training provides an annotation efficient route to holistic three dee scene understanding, which is what we need for real world applications where data collection is hard <ref:2506.17545#pg1>.

Meng: And I think if this approach scales well and handles the complexities of dynamic environments better than they currently show, it could become a viable method for deploying complex spatial awareness into embodied agents.

Lalam: The final conclusion here is that Scene-R1 demonstrates that video centric perception coupled with reinforcement learning based reasoning offers a practical route to transparent and scalable three dee scene understanding without the cost of dense three dee annotation <ref:2506.17545#pg1>.

Tom: It’s a way to get high accuracy in three dee reasoning while providing clear, interpretable chains of thought, which is what makes this work really useful for the community right now <ref:2506.17545#pg1>.

Zhihao Yuan, Shuyi Jiang, Chun-Mei Feng, Yaolun Zhang, Shuguang Cui, Zhen Li

FNii-Shenzhen Institute of High Performance Computing, A*STAR Singapore, Singapore University of Technology and Design

cs.CV

Submitted: 2025-06-21

Updated: 2026-10-20

Code: https://github.com/819346675/Scene-R1

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist: Scene-R1 introduces a video-grounded framework that learns to reason about 3D scenes without any point-wise 3D instance supervision by pairing reinforcement-learning-driven reasoning with a

Key concepts

Temporal Grounding: This is the first stage where the vision-language model analyzes a video query to select the most relevant video segments. It helps narrow down which parts of the video contain information needed to answer a question, creating a focused context for subsequent reasoning steps.
Image Grounding with SAM2: After temporal selection, this stage grounds the target object within individual frames. The model predicts 2D bounding boxes in text, which are then used with SAM2 (a segmentation module) to generate precise pixel masks. These masks are then projected back into 3D space for accurate localization.
Reinforcement Learning Objectives: The model is trained using Group-Relative Policy Optimization (GRPO) instead of traditional actor-critic methods. The reward system is complex, involving metrics like IoUtimeLˆ,L⋆ and format rewards to guide the LLM toward generating correct reasoning steps and accurate 3D outputs.
Annotation-Efficient 3D Grounding: Scene-R1 achieves high performance without requiring dense 3D point-wise labels. Instead, it relies on task-level supervision, such as sparse 2D bounding boxes or textual labels. This makes the method practical for real-world applications where expensive 3D annotation is unavailable.

Terminology

Grounding — 3× in this paper · explained in 3 episodes
SAM2 — 1× in this paper · explained in 2 episodes

Summary

The gist: Scene-R1 introduces a video-grounded framework that learns to reason about 3D scenes without any point-wise 3D instance supervision by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline.

Scene Overview and Motivation

Existing 3D-aware LLMs act as black boxes because they output bounding boxes or textual answers without revealing how those decisions are made, and they still rely on pre-trained 3D detectors to supply object proposals<ref:2506.17545#pg2>. Scene-R1 addresses this by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline<ref:2506.17545#pg3>. This approach eliminates the need for 3D detector-based proposals while capturing fine geometry and material cues through tracking objects using SAM2 to produce pixel-accurate masks in RGB frames, which are then projected back into 3D<ref:2506.17545#pg3>. The training pipeline only needs task-level 2D boxes or textual labels without dense 3D point-wise labels<ref:2506.17545#pg3>. Scene-R1 surpasses existing open-vocabulary baselines on multiple datasets while delivering transparent, step-by-step rationales<ref:2506.17545#pg3>.

Framework Architecture

Scene-R1 is organized into two grounding stages that share the same VLM backbone but are fine-tuned under distinct RL objectives<ref:2506.17545#pg3>. The first stage is Temporal Grounding, where the vision language model selects query-relevant RGB snippets and generates explicit chain-of-thought rationales<ref:2506.17545#pg5>. In the subsequent Image Grounding stage, the VLM grounds the target object on each frame of the selected snippets, directly generating corner coordinates of 2D bounding boxes in text format<ref:2506.17545#pg5>. These 2D boxes are then fed along with detected bounding boxes to a prompt-based segmenter (SAM2) to produce pixel-accurate masks, which are then projected back into 3D<ref:2506.17545#pg5>.

Reinforcement Learning Objectives

The training leverages Group-Relative Policy Optimization (GRPO) to optimize the LLM, which departs from actor–critic methods by using sampling and group-normalized rewards<ref:2506.17545#pg4>. The reward calculation involves two main components for temporal grounding: IoUtimeLˆ,L⋆ and a format reward rform(o) which checks if the output o matches the required template<ref:2506.17545#pg5>. For image grounding, the reward rIG is composed of IoUboxbˆτ, b⋆ τ, a JSON-specific reward rjson(o) to ensure valid JSON output, and rform(o)<ref:2506.17545#pg5>. The final 3D grounding output is obtained by aggregating points across the temporal window Sˆ using the tightest axis-aligned bounding box that encloses P<ref:2506.17545#pg5>.

Key Contributions and Results

The approach yields two central contributions: RL-generated transparent reasoning, which is the first 3D framework to generate and expose chain-of-thought rationales through R1-style reinforcement learning<ref:2506.17545#pg3>, and annotation-efficient 3D grounding, which achieves performance competitive with detector-based pipelines on standard 3D indoor understanding benchmarks without dense 3D instance labels<ref:2506.17545#pg3>. Quantitative analysis shows that Scene-R1 surpasses the previous best label-free baseline, OpenScene [32], by +25.6 and +10.6 percentage points in 3D Visual Grounding<ref:2506.17545#pg7>. Furthermore, under constraints where only weak rewards are used for 3D Visual Question Answering (3DVQA), Scene-R1 attains an average EM-1 of 49.4%, outperforming the official SQA3D baseline (46.6%)<ref:2506.17545#pg9>.

Qualitative Demonstrations

Qualitative results illustrate the method's versatility across distinct 3-D reasoning tasks<ref:2506.17545#pg10>. In 3D Visual Grounding, Scene-R1 selects frames that cover all referenced objects and overlaps the ground-truth segment with a temporal IoU of 0.71<ref:2506.17545#pg10>. For affordance grounding, the model highlights the precise actionable sub-region, such as a keypad for dialing numbers or a specific drawer for opening<ref:2506.17545#pg10>. The generated chain-of-thought explains the spatial hierarchy in natural language, providing human-readable evidence for the model’s decision<ref:2506.17545#pg10>.

Conclusion

Scene-R1 demonstrates that video-centric perception, coupled with reinforcement-learning-based reasoning, offers a practical route to transparent and scalable 3D scene understanding without the cost of dense 3D annotation<ref:2506.17545#pg3>. This method produces faithful, interpretable chains of thought and shows that RGB-D video, paired with reinforcement-learning post-training, offers a practical and annotation-efficient route to holistic 3D scene understanding<ref:2506.17545#pg10>.

Limitations

The approach inherits both its strengths and weaknesses from the foundation models on which it is built, i.e. Qwen 2.5-VL and SAM-2<ref:2506.17545#pg11>. It is still behind fully supervised, detector-centric systems in absolute accuracy, and extending the method to outdoor and dynamic environments remains future work<ref:2506.17545#pg11>. Because Scene-R1 learns from large-scale web data, it can inadvertently amplify societal biases present in those corpora, and the substantial computational demand of training and inference carries a non-trivial carbon footprint<ref:2506.17545#pg11>.

References

[1] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In European Conference on Computer Vision, pages 422–440. Springer, 2020<ref:2506.17545#pg12>.

[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025<ref:2506.17545#pg12>.

[6] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202–221. Springer, 2020<ref:2506.17545#pg12>.

[7] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than 3. https://github.com/Deep-Agent/R1-V, 2025. Accessed: 2025-02-02<ref:2506.17545#pg12>.

[8] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in Neural Information Processing Systems, 35:20522–20535, 2022<ref:2506.17545#pg12>.

[9] Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Gang Yu, and Tao Chen. End-to-end 3d dense captioning with vote2cap-detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11124–11133, 2023<ref:2506.17545#pg10>.

[10] Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, YU Gang, Taihao Li, and Tao Chen.

Improvements for AI systems

Improve model interpretability by generating transparent, step-by-step rationales for 3D reasoning, as Scene-R1 is to our knowledge, the first 3D framework that generates and exposes chain-of-thought rationales through R1-style reinforcement learning.
Achieve annotation efficiency by eliminating the need for dense 3D point-wise labels; Scene-R1 performs reasoning without any point-wise 3D instance supervision and only requires task-level 2D boxes or textual labels without dense 3D point-wise labels.
Enable fine geometry and material cue capture by using a two-stage grounding pipeline where the second stage analyzes the image to predict bounding boxes, which are then used with SAM2 to produce pixel-accurate masks in RGB frames, and project them back into 3D.
Enhance robustness to reward scaling by employing Group-Relative Policy Optimization (GRPO) which converts raw rewards into relative scores, making the optimization invariant to the reward scale and emphasizes responses that outperform their peers.
Improve temporal localization accuracy by using a reward function combining frame-level IoU and format adherence: rTG = IoUtimeLˆ,L⋆ + λ rform(o), λ = 0.1, which incentivizes the model to select the video snippets most relevant to an open-ended query.
Improve 3D visual question answering (3DVQA) by using a reward formulation that rewards factual accuracy and clarity: rQA = EM(a, a⋆) + EM-R(a, a⋆) + λ rform(o), allowing the model to answer questions without ever consulting dense 3D annotations.