Learning to Think and Explore in 3D World

summary

Video file (mp4)

The gist

The gist: Act3D enables vision-language models to learn to think and explore in 3D space by treating 3D representations as interactive environments, allowing them to actively acquire task-relevant

In short

Act3D enables vision-language models to think and explore in 3D space by treating 3D representations as interactive environments. The model learns a policy by alternating between reasoning and interacting with the environment, acquiring visual evidence through movement. This outcome-driven reinforcement learning allows the model to learn not just how to reason over existing views, but also which new views it needs to acquire to answer a question.

Key concepts

Policy Rollout
This is the complete sequence of actions and observations a model takes during one interaction episode. The model alternates between reasoning about what it knows and taking an action (like moving or turning) to gather new visual information. This entire sequence constitutes one 'rollout' that the model uses to learn how to think and explore effectively.
Partial Observability (POMDP)
The environment is modeled as a Partially Observable Markov Decision Process, meaning the model does not know the complete state of the 3D world at any given moment. The true state includes where the camera is and which direction it's facing. Instead, the model forms an internal 'belief' about these hidden states based only on its rendered views and movement history.
Outcome-Driven Reinforcement Learning (GRPO)
The training uses Group Relative Policy Optimization (GRPO) where the reward is primarily based on whether the final answer is correct. This method optimizes the policy by assigning advantages to all tokens in a trajectory, teaching the model which interactions lead to better outcomes. Crucially, it forces the model to learn an effective exploration strategy based on performance improvement rather than being told exactly what to explore.
Relative Parameterization
The navigation actions use relative parameterizations like 'move left' or 'turn right.' This means the movement commands are scale-invariant and camera-centric, making them intuitive for the model. This contrasts with absolute actions that require knowing a specific global reference frame, allowing the model to navigate naturally within its current 3D view.

Terminology used across episodes

This episode discusses

The paper

Learning to Think and Explore in 3D World · Read on arXiv

Yijiang Li, Bingyang Wang, Nuno Vasconcelos

University of California San Diego · Georgia Institute of Technology

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Learning to Think and Explore in 3D World".

Jane: The gist: Act3D enables vision-language models to learn to think and explore in 3D space by treating 3D representations as interactive environments,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We’re talking about "Learning to Think and Explore in three dee World <ref:2610.21783#pg1,Learning to Think and Explore in 3D World>." The authors are Yijiang Li, Bingyang Wang, and Nuno Vasconcelos from UC San Diego and Georgia Tech.

Jane: Those are solid academic folks. So what's the main takeaway from this specific paper? What is the big idea they’re pushing here?

Tom: They’re focusing on how vision-language models can learn to think and actively explore in a three-dimensional space by treating those three dee representations as interactive environments <ref:2610.21783#pg1,can learn to think and actively explore in>.

Lu: Essentially, they are changing the input assumption; instead of giving the model all the data upfront, they keep the three dee scene external and let the model interact with it through actions like moving or changing its viewpoint to gather what it needs <ref:2610.21783#pg1>.

Meng: So, if a model is trying to figure out where something is in a room, instead of just being handed a picture of that room, it learns how to walk around and look from different angles.

Lalam: Exactly. They are modeling the whole process as alternating between reasoning and interaction, starting from an initial first-person view and a question the model has.

The paper's summary: Tom: So, the summary shows they establish this learning process by alternating between reasoning and interaction, starting with a specific first-person view and a visual question.

Jane: How does that iteration work? What’s the actual sequence of things the model does in one round?

Tom: The model starts by reasoning over what it sees, then takes an action—maybe moving or turning—which changes its view, and then it observes that new view before reasoning again. This repeats until it commits to a final answer.

Lu: They outline a rollout trajectory where the model alternates between reasoning and interaction, iteratively acquiring observations through navigation and viewpoint control before committing to the final answer.

Meng: It sounds like they're designing a policy rollout where the model jointly decides how to get new visual information, how to move in space, and how to reason about what it already has seen.

Lalam: They call this entire episode of interaction a policy rollout, meaning the model learns the whole sequence—the acquisition of visual information, the spatial interaction, and the reasoning—all together.

The paper's improvements: Tom: Now let’s look at how they improved upon existing methods. They use a two-stage training procedure first.

Jane: What does that two-stage setup involve? Is it just one training phase followed by another, or something more specific?

Tom: First, they do trajectory-supervised fine-tuning to set up the interaction protocol using some curated interaction demonstrations. That gets an initial coherent reason–act–observe policy in place.

Lu: This supervised fine-tuning step is important because it initializes a coherent reason–act–observe policy rather than just prescribing a single way for the model to explore; it’s about learning the protocol first.

Meng: I like that idea of setting up the rules before optimizing everything else, but what happens in the second stage? That’s where the real learning should be.

Tom: Second, they apply reinforcement learning with verifiable reward to optimize that full interactive policy using rewards driven primarily by final-answer correctness.

Lu: The key improvement here is that reinforcement learning provides no direct supervision over which viewpoints to visit or which actions to take; instead, the model has to learn an effective exploration policy solely from how much its interactions improve performance.

Lalam: So they’re using outcome-driven reinforcement learning where the reward is mainly based on getting the final answer correct, and that reward guides it toward better interaction strategies without telling it exactly which path to take.

Conclusion: Tom: To wrap up, this whole framework shows that outcome-based reinforcement learning lets vision-language models not only reason over visual evidence but also adaptively acquire the evidence needed for three-dimensional reasoning.

Jane: So, the main implication is that these models can learn when and how to actively seek out the visual data they need to make a question answerable in three dee space <ref:2610.21783#pg1>.

Lu: This framework shows that by treating three dee representations as interactive environments, we enable VLMs to learn not just about observed evidence but also how to acquire it <ref:2610.21783#pg1>.

Meng: If this works well in practice, it means we can build AI systems that can handle complex spatial queries where the answer isn't available in a single snapshot.

Lalam: And they show that Actthree dee achieves an average accuracy of fifty-three point two percent across six three dee benchmarks, which is a solid result for this kind of interactive learning setup.

Tom: That's the gist of "Learning to Think and Explore in three dee World <ref:2610.21783#pg1,Learning to Think and Explore in 3D World>." It’s about learning the exploration policy itself through interaction with the environment.

More episodes

← Home