Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation

arXiv:2610.21850 · cs.CV, cs.GR · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Seeing Both the Forest and the Trees".

Jane: The gist: STRUCTPARTS is a framework that allows part-decomposed generation models to inherit both geometric detail and structural coherence from holistic 3D priors through dual-frame coordinate interleaving.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're diving into "Seeing Both the Forest and the Trees: Reusing Holistic three dee Priors for Part-Decomposed Generation <ref:2610.21850#pg1,Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for>." The authors are Zhao, Huang, Duan, and Zhao. They’re proposing a way to combine geometric detail with structural coherence in part-decomposed generation.

Jane: It's about taking those two properties—the fine detail inside each part and the overall structure of the whole object—and making a model capable of producing both simultaneously using this new mechanism they call STRUCTPARTS.

Lu: The implication is that we don't need to train a part-specific model from scratch; we can adapt an existing holistic prior by just reinterpreting how it’s processed locally and globally, which saves a ton of training time.

Meng: I’m still focused on the structural coherence. If the interleaving isn't right, we risk getting parts that look good individually but don't assemble into a coherent object when we try to put them together later. What exactly is this mechanism doing differently?

Jane: The core technical contribution is the dual-frame coordinate interleaving mechanism, which lets each shape latent be interpreted under both a part-local and an object-global reference frame at the same time.

Tom: So, it’s not just one frame; it’s two frames for every shape latent. And this dual interpretation happens through a global-local interleaved transformer that alternates between intra-part and inter-part attention.

The paper's summary: Jane: So, to summarize the paper, they are addressing the fact that recent image-conditioned three dee generation models already have structured representations that capture both high fidelity global structure and local geometric detail <ref:2610.21850#pg1,recent image-conditioned 3D generation models>.

Tom: But the challenge is extending those priors to part-decomposed generation because those priors were never trained for parts. The authors show how STRUCTPARTS solves this by using this dual-frame mechanism to inherit both required properties from the holistic prior.

Lu: It’s essentially treating each part as a complete shape locally, while also coordinating them as an anchored assembly globally, which is what they call reasoning at both scales simultaneously.

Meng: I see that connecting the geometric detail on a local level with the structural coherence on a global level is exactly what we need for any good three dee reconstruction task <ref:2610.21850#pg1>. It sounds like they’ve found a way to keep those two things balanced during generation.

Jane: They set up their input using this segmented mesh scaffold, which combines a coarse mesh from an off-the-shelf holistic model with a segmentation mask that stays within the prior’s native three dee representation <ref:2610.21850#pg1>.

Tom: So, the scaffold is essentially a bridge; it’s something the holistic prior can read without any objective shift, and it's also a byproduct of those upstream generation stages.

The paper's improvements: Jane: The improvements they suggest focus on how this method handles input and output. They use this dual-frame approach to ensure that in the intra-part attention layers, where F equals L and Zkv equals Z, each part is processed as a standalone shape, reusing the prior’s geometric detail exactly as it was during pretraining.

Tom: And then in the inter-part attention layers, where F equals G and Zkv equals Z, the parts are coordinated as one anchored assembly, which preserves the prior’s structural coherence across the whole object.

Lu: That alternating attention structure is smart because it lets them leverage both strengths of the prior at different stages of processing. It ensures they get the local detail when they need it and maintain global structure when coordinating.

Meng: So what does this mean practically for deployment? Does this complex interleaving add too much computational overhead to run on, say, a production server?

Tom: They show that on PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models with substantially lower training costs. That’s a big win when you think about the efficiency gains for running these things.

Conclusion: Jane: To wrap up, the paper with this title, "Seeing Both the Forest and the Trees: Reusing Holistic three dee Priors for Part-Decomposed Generation," shows that we can inherit both geometric detail and structural coherence from a holistic prior using only lightweight finetuning <ref:2610.21850#pg1,Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for>.

Tom: The main implication is that they’ve created a framework that naturally supports part-mesh refinement when the scaffold is coarse, and also allows for training-free merge and split operations through segmentation regrouping.

Lu: This lowers the barrier for nonexperts because you can iterate on three dee content without needing to retrain a part-specific model from scratch; you just have to reinterpret the prior <ref:2610.21850#pg1>.

Meng: I’m still thinking about the robustness. They mentioned that whole-shape performance stays close to clean at one percent geometry noise, which is important for real-world data where things aren't perfect.

Lalam: From my perspective, I see this as a way to make three dee assets much more flexible and controllable through these training-free editing operations, which is a really practical application for generative models <ref:2610.21850#pg1>.

Jane: It’s definitely a significant step in making part-decomposed generation more accessible by cleverly leveraging existing knowledge from holistic models. That’s where we'll leave it for this one.

Yiqun Zhao, Binbin Huang, Haobin Duan, Zibo Zhao, Shenghua Gao

University of Hong Kong · Transengram · Tencent · Shenzhen Loop Area Institute

cs.CV, cs.GR

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: NeurIPS 2026, project page: https://zhao-yiqun.github.io/structparts/

Project page: https://zhao-yiqun.github.io/structparts

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: The gist: STRUCTPARTS is a framework that allows part-decomposed generation models to inherit both geometric detail and structural coherence from holistic 3D priors through dual-frame coordinate

Key concepts

Dual-Frame Coordinate Interleaving: This mechanism interprets each shape latent using two coordinate systems simultaneously: one local to the specific part and one global for the entire object. This forces the model to reason at both scales at once, treating a single part as both a complete shape locally and a component within a larger forest globally.
Segmented Mesh Scaffold: This is an input structure created by combining a coarse mesh from an existing holistic 3D model with segmentation masks derived entirely from the prior's native representation. This scaffold serves as the structural guide that the holistic prior reads without any objective shift, providing necessary information for part-level processing.
Global-Local Interleaved Transformer: This transformer alternates between two types of attention: intra-part attention (where each part is treated as a standalone shape to reuse geometric detail) and inter-part attention (where parts are coordinated as an assembly to preserve structural coherence across the whole object.
Part-Mesh Refinement: The framework can recover fine geometric details for individual parts even when the initial scaffold input is coarse. This is achieved because the dual-frame embeddings retain image conditions in every block, allowing the model to synthesize high-fidelity geometry from lower-resolution structural guidance.

Terminology

Coherence — 3× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist: STRUCTPARTS is a framework that allows part-decomposed generation models to inherit both geometric detail and structural coherence from holistic 3D priors through dual-frame coordinate interleaving.

How it works

The core technical contribution of STRUCTPARTS is a dual-frame coordinate interleaving mechanism where "each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local interleaved transformer that simultaneously reuses the prior’s geometric detail at the part scale and its structural coherence at the whole-object scale (Page 1). This mechanism forces the model to reason at the part scale and the object scale at once," treating each part as both a complete shape locally and as part of a larger forest globally (Page 2).

The framework utilizes a segmented mesh scaffold as its structural input, which is constructed by combining a coarse mesh from an off-the-shelf holistic 3D generation model with a face-level segmentation mask kept entirely within the prior’s native 3D representation (Page 2). This segmented mesh scaffold serves as a structural input the holistic prior reads without any objective shift, and one that off-the-shelf upstream stages produce as a by-product of holistic generation and segmentation (Page 2).

Dual-Frame Coordinate Interleaving

For each part, the model samples dense points on its scaffold surface to construct voxel indices in two complementary coordinate frames: local frame normalized within each part, and a global frame normalized across the whole object (Page 3). These indices are then used to compute two distinct Rotary Positional Embeddings (RoPEs): one for each part, RoPEL(x), and one for the object, RoPEG(x) (Page 3).

These dual frames guide the processing through a global-local interleaved transformer that alternates between intra-part and inter-part attention (Page 3). In the intra-part attention layers, where F = L and Zkv = Z(k), each part is processed as a standalone shape, reusing the prior’s geometric detail exactly as during pretraining (Page 3). In the inter-part attention layers, where F = G and Zkv = Z, the parts are coordinated as one anchored assembly, preserving the prior’s structural coherence across the whole object (Page 3).

Training and Inference

STRUCTPARTS is trained using a flow-matching objective (LFM) supervised by predicting the velocity from a noisy latent state, conditioned on both time step t and image features Cimg (Page 3). During inference, scaffolds come from a single image via structure generation [13] and 3D segmentation models [14, 15] (Page 3). The dual-frame embeddings only require coarse voxel indices from the scaffold, and the image condition is retained in every block, enabling part-mesh refinement even when the scaffold is coarse (Page 3).

Key Results and Applications

The method demonstrates superior performance across different evaluation protocols. In layout-conditioned evaluation (part-level), STRUCTPARTS achieves the best decomposition quality across all metrics, with qualitative comparisons in Fig. 3 (Page 4). In image-to-decomposed-parts evaluation (shape-level), STRUCTPARTS is comparable to released X-Part and outperforms the other baselines by a clear margin (Page 4).

Furthermore, the framework supports two key downstream capabilities without additional training:

Part-mesh refinement from a coarse scaffold, where it synthesizes fine geometric details that faithfully match the input image (Page 5).
Part-level editing via scaffold regrouping, which supports training-free merge and split operations by relabeling faces in the segmentation mask L (Page 5).

Efficiency and Robustness

StructPARTS shows efficiency gains, matching or surpassing prior part-decomposed models at a fraction of their training cost on PartObjaverse-Tiny (Page 1). Ablation studies confirm that the dual frame design is necessary, as variants like LRoPE only severing it from the holistic prior and producing incoherent global structure (Page 8). The method also shows robustness to scaffold errors, with whole-shape performance remaining close to clean at 1% geometry noise (Page 8).

Conclusion

STRUCTPARTS successfully inherits both the geometric detail and structural coherence of a holistic 3D prior with only lightweight finetuning, while naturally supporting part-mesh refinement and part-level editing (Page 5). This approach lowers the barrier for nonexperts to iterate on 3D content by enabling training-free editing operations (Page 18).

Improvements for AI systems

Part-decomposed generation inherits both geometric detail and structural coherence from a holistic prior through dual-frame coordinate interleaving, allowing it to read the prior at both scales simultaneously. This enables models to produce assets that possess both properties by processing each part as a complete shape locally while coordinating them as an anchored assembly globally.
The system can perform low-fidelity scaffold refinement, as the method can recover detail directly from the image even when the input scaffold is coarse, stating that it enables part-mesh refinement even when the scaffold is coarse.
Part-level editing becomes training-free by leveraging a segmented mesh scaffold where users can perform merge and split operations through segmentation regrouping in a trainingfree manner, allowing them to control decomposition granularity at inference time.

Abstract

Generating 3D assets with explicit parts requires two properties at once: geometric detail inside each part and structural coherence across the whole object. Recent image-conditioned 3D generation models adopt a structured 3D representation that anchors tokens to explicit spatial positions, capturing both properties, but their output remains a single fused mesh. We ask how a part-decomposed generation model can inherit both properties of these holistic priors, and answer with STRUCTPARTS, a framework whose core technical contribution is a dual-frame coordinate interleaving mechanism: each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local inter-leaved transformer that simultaneously reuses the prior's geometric detail at the part scale and its structural coherence at the whole-object scale. As its structural input, STRUCTPARTS uses a segmented mesh scaffold, a coarse mesh paired with a face-level segmentation mask kept entirely within the prior's native 3D representation. With only lightweight finetuning, both intra-part detail and inter-part coherence emerge from the same pretrained weights. On PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models at a fraction of their training cost, while naturally supporting part-mesh refinement and part-level editing.