Learning to Think and Explore in 3D World

The gist: Act3D enables vision-language models to learn to think and explore in 3D space by treating 3D representations as interactive environments, allowing them to actively acquire task-relevant visual evidence through interaction with the environment and reasoning over accumulated evidence.

How it works

The model learns a policy rollout by alternating between reasoning and interaction, starting from an initial first-person view and a visual question, iteratively acquiring observations through navigation and viewpoint control before committing to a final answer The entire episode of interaction constitutes a policy rollout, in which the model jointly determines how to acquire visual information acquisition, spatial interaction, and reasoning.

The training procedure employs a two-stage approach:

1. Trajectory-supervised fine-tuning is performed first to establish the interaction protocol using curated interaction demonstrations This initializes a coherent reason–act–observe policy rather than prescribing a unique exploration strategy.

2. Reinforcement learning with verifiable reward is then applied to optimize the full interactive policy using rewards driven primarily by final-answer correctness Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take, instead requiring the model to learn an effective exploration policy solely from the extent to which its interactions improve performance.

Reasoning in 3D under partial observability

The reasoning process is formalized as a partially observable Markov decision process (POMDP) where the environment state is defined by the camera position and viewing direction, and the policy observes rendered egocentric views. The action space consists of two parameterizations: relative navigation actions that allow for intuitive movements like “move left” or “turn right,” and absolute parameterizations that name a target pose directly. The policy conditions on the complete interaction history to form an implicit belief over the accumulated observations.

The objective function to be optimized is the expected finite-horizon, undiscounted return, which jointly evaluates the policy’s ability to acquire informative observations and reason over them to produce the correct answer. The reward function R(τ) is a weighted sum of answercorrectness, interaction, and output-format terms. The interaction reward is credited only in rollouts that end in a correct answer and treats verified grounding as a shortcut to the full credit rather than as a separate bonus.

Outcome-driven reinforcement learning

The model optimizes the policy using Group Relative Policy Optimization (GRPO) by standardizing the outcome reward to obtain an advantage Aˆi. This trajectory-level advantage is assigned to all model-generated tokens in τi, allowing the policy to learn not only how to reason over acquired evidence but also which evidence to acquire in the first place. The system evaluates performance across six 3D benchmarks, achieving an average accuracy of 53.2% on ACT3D.

Learning dynamics and exploration behavior

RL training progressively encourages more extensive interaction with the environment, as evidenced by the increase in the average number of interactions per episode from approximately 3.3 to 8.3 during mixed-data training. Adaptive exploration behavior emerges, where performance gains are generally larger for questions requiring more exploration, increasing from 0.6% in the lowest-demand group to 16.3% in the highest. The policy learns to search for occluded targets and avoids unnecessary exploration when relevant evidence is already visible.

Initial observation and interface design

The model is initialized with a single 640 × 480 render of the scene, demonstrating that additional initial visual context does not necessarily improve performance. Act3D formulates reasoning as interaction with a 3D scene representation, making the design of the visual observation interface an important consideration. Rendered observations consistently outperform real-image retrieval across all 13 evaluation checkpoints under matched training conditions.

Conclusion

Act3D demonstrates that outcome-based RL enables VLMs not only to reason over visual evidence but also to adaptively acquire the evidence needed for 3D reasoning. This framework shows the potential of learning to think and explore in 3D world by treating 3D representations as interactive environments rather than assumed complete inputs. The resulting policy learns how to acquire visual evidence needed for 3D reasoning. This demonstrates that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The findings demonstrate the potential of outcome-based RL to learn not only how to reason over visual evidence, but also how to actively acquire the evidence needed for 3D reasoning. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable. This framework shows that Act3D achieves the highest average accuracy of 53.2% across six 3D benchmarks. The findings demonstrate that RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. The resulting learning problem is not merely to reason over observed evidence, but to determine which observations should be acquired to make the question answerable <ref:2610.

1. Bold header: Outcome-Driven Reinforcement Learning for Adaptive Exploration

By applying reinforcement learning with rewards driven primarily by final-answer correctness, Act3D learns how to acquire it [visual evidence] by optimizing a policy solely from the extent to which its interactions improve performance, enabling it to learn when and how to acquire it rather than relying on fixed exploration strategies.

2. Bold header: Adaptive Exploration Based on Information Demand

The system exhibits adaptive behavior where Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration, as it learns to search for occluded targets, avoid unnecessary exploration, and acquire complementary viewpoints for spatial comparisons.

3. Bold header: Robust Policy Initialization via Supervised Fine-Tuning

The two-stage training procedure utilizes trajectory-supervised fine-tuning first establishes the interaction protocol, which initializes a coherent reason–act–observe policy before reinforcement learning optimizes the full interactive policy, suggesting that SFT facilitates adaptation to the interactive setting and may accelerate subsequent RL optimization.

4. Bold header: Consistency Set Formulation for Answerability

The model can determine if a question is answerable by checking the consistency set, as q is answerable from ht when all scenes consistent with the acquired observations induce the same answer, allowing it to progressively refine the set of compatible scenes beyond what a passive model can achieve.

5. Bold header: Action Space Efficiency via Relative Parameterization

Act3D adopts a relative parameterization for navigation actions, which are scale-invariant, camera-centric, and map naturally to interpretable motions such as 'move left' or 'turn right', contrasting with absolute actions that require reasoning in a scenedependent global reference frame.

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.