SUPRA-TEX: 3D Generation with Super-Aligned Texture

The gist Single-image 3D generation aims to dress a given 3D mesh with relightable materials while faithfully preserving the visual details of a reference image The method introduces SUPRA-Tex, built on the principle of decoupling visible-material estimation, surface alignment, and observation-guided 3D completion.

How it works

The framework is built on the division of labor between a 2D generative prior and the 3D generator to handle the complex task of generating relightable PBR materials <ref:2610.22352#pg2> Specifically, SUPRA-Tex first trains an MLLM-based image editing model to predict lighting-separated PBR maps in the reference view in a single forward pass <ref:2610.22352#pg3>. These predicted maps are then projected onto the visible surface and encoded with a pretrained material VAE encoder into native observation latents that share the generation target’s latent space <ref:2610.22352#pg3>.

How it works (Cont.)

An interleaved flow transformer holds these latents as fixed, read-only anchors while alternating unobserved region and full-surface updates to generate complete material latents <ref:2610.22352#pg3>. This design achieves super-alignment with the reference, delivering What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces <ref:2610.22352#pg3>. The generator is trained solely on public datasets, achieving SOTA performance in both geometry-conditioned texturing and end-to-end asset generation <ref:2610.22352#pg3>.

How it works (Cont.)

The process involves several key stages:

1. Reference-view material estimation, where an MLLM model predicts lighting-separated PBR maps <ref:2610.22352#pg3>.

2. Surface-aligned native observation encoding, where these maps are projected onto valid surface locations and encoded into native observation latents using the pretrained material VAE encoder <ref:2610.22352#pg4>.

3. Observation-guided 3D completion, where an interleaved flow transformer generates complete material latents conditioned on geometry latents and the native observation representation <ref:2610.22352#pg5>.

How it works (Cont.)

The native observation representation Co is structured as a set of tuples containing the latent cell index, its encoded latent vector, coverage fraction ρs, and boundary distance ds <ref:2610.22352#pg5>. This structure encapsulates native material observations alongside their physical extent without fictitious padding <ref:2610.22352#pg5>. The interleaved flow transformer uses two streams: a dynamic generation stream X tracking the hidden state of Zt over S, and a static observation stream O serving as immutable reference memory over So <ref:2610.22352#pg5> The query coordinates alternate across blocks, alternating between unobserved cells U and the full surface S to concentrate generative capacity on unobserved surfaces while maintaining global coherence <ref:2610.22352#pg6>.

How it works (Cont.)

The training objective is defined by the flow matching loss LFM = E[MSE(vθ(Zt, t; O, G), v∗)] averaged over all valid tokens and channels, including observed coordinates <ref:2610.22352#pg6>. The observation representation remains fixed and is not noised along the flow path, serving as conditioning input to the generator <ref:2610.22352#pg6>.

How it works (Cont.)

In inference, the process starts from Z1 ∼ N (0, I) over S and integrates the learned velocity field with Euler steps to decode final latents with Dmat into full-surface PBR attributes for export <ref:2610.22352#pg6>.

How it works (Cont.)

The method is evaluated on two datasets, DTC-Tex and Complex-Tex, using metrics such as PSNRfg, SSIM, LPIPS, CLIP-I, and FDDINOv3 <ref:2610.22352#pg7> SUPRA-Tex achieves the best mean in every reported metric on both view splits and datasets <ref:2610.22352#pg9>. The results show that SUPRA-Tex leads across all shaded and PBR metrics on both datasets <ref:2610.22352#pg9>.

How it works (Cont.)

The method is trained solely on public datasets, and with synthetic back-view conditioning, it achieves reference fidelity and visual quality comparable to or better than those of the leading commercial systems in our comparisons <ref:2610.22352#pg3> The key insight is to establish surface-aligned material observations before 3D generation, allowing the generator to focus on preserving reference details and completing unobserved surfaces <ref:2610.22352#pg3>.

How it works (Cont.)

The method demonstrates that predicted MR improves shaded PSNRfg by 0.44 and 0.26 dB on DTC-Tex, with lower MR MAE on both datasets <ref:2610.22352#pg9> Replacing all predicted PBR channels with ground truth further improves shaded PSNRfg by 2.31/1.12 dB and albedo PSNRfg by 2.62/2.51 dB on DTC-Tex/Complex-Tex <ref:2610.22352#pg9>.

How it works (Cont.)

The method is also shown to handle image editing, where the image editing model can supply appearance for surfaces the reference never shows, or change the appearance of surfaces it does <ref:2610.22352#pg3> Synthetic back-view conditions allow for generating geometry-consistent back views and encoding them as an additional material condition <ref:2610.22352#pg3>.

How it works (Cont.)

The framework is limited by large image–geometry discrepancies, where geometry predicted by Pixal3D⋆ can place projected material observations on incorrect surface locations <ref:2610.22352#pg2> Hallucinated geometry on unseen surfaces can hinder coherent texture completion because SUPRA-Tex conditions on fixed geometry <ref:2610.22352#pg2>. Inconsistent completion across visibility boundaries is a challenge when text or patterns extend beyond the visible region, potentially leading to inconsistent text or patterns across the visibility boundary <ref:2610.22352#pg3> Errors in 2D material estimation can lead to implausible reflections under rendering if they condition the 3D generator <ref:2610.22352#pg2>.

How it works (Cont.)

SUPRA-Tex achieves sampling speedups of 1.20× on DTC-Tex and 1.18× on Complex-Tex <ref:2610.22352#pg7> Observation KV caching adds speedup, and the interleaved schedule reduces query-token evaluations by 17.54% and 16.91% versus querying every cell at each block <ref:2610.22352#pg7>. The method is competitive with commercial systems in human evaluations on the mixed real-and-synthetic comparison set <ref:2610.22352#pg3>.

How it works (Cont.)

The method is trained solely on public data, and with synthetic back-view conditioning, it is also competitive with commercial systems in human evaluations on the mixed real-and-synthetic comparison set <ref:2610.22352#pg3>. The results demonstrate the value of spatially aligned, directly usable material observations for reconciling reference fidelity with full-surface generation <ref:2610.22352#pg2>.

How it works (Cont.)

The method is trained using assets from TexVerse, Objaverse-XL, ABO, HSSD, and open synthetic 3D collections <ref:2610.22352#pg3> The 2D material estimator F2D is fine-tuned on aligned RGB images and ground-truth material and normal maps rendered from publicly available 3D assets <ref:2610.22352#pg6>. The training objective uses the flow-matching objective in Eq. (10) averaged over all valid tokens and channels <ref:2610.22352#pg6>.

How it works (Cont.)

The method is trained using a global batch size of 576 using AdamW, with a global learning-rate schedule involving linear warmup and cosine decay <ref:2610.22352#pg6> The final training stage uses 64 GPUs and FlashAttention-3 acceleration <ref:2610.22352#pg6>.

How it works (Cont.)

SUPRA-Tex achieves the best mean in every reported metric on both view splits and datasets <ref:2610.22352#pg9> The results show that SUPRA-Tex leads across all shaded and PBR metrics on both datasets <ref:2610.22352#pg9>. The method is trained solely on public datasets, and with synthetic back-view conditioning, it achieves reference fidelity and visual quality comparable to or better than those of the leading commercial systems in our comparisons <ref:2610.22352#pg3>.

How it works (Cont.)

The key insight is to establish surface-aligned material observations before 3D generation, allowing the generator to focus on preserving reference details and completing unobserved surfaces <ref:2610.22352#pg3> The method is trained solely on public datasets, and with synthetic back-view conditioning, it achieves reference fidelity and visual quality comparable to or better than those of the leading commercial systems in our comparisons <ref:2610.22352#pg3>.

**How it works (Cont.

1. Bold header: Native Observation Encoding for Grounding

This improvement allows 3D generation to anchor directly on established material evidence by projecting visible reference evidence into native material observations, which simplifies learning by allowing the model to use physical PBR observations to focus on reference preservation and unobserved surface completion.

2. Bold header: Interleaved Flow Transformer for Coherent Completion

The system utilizes an interleaved flow transformer that reads the native observation latents as fixed, read-only reference memory at every block, alternating between unobserved region and full-surface updates to generate complete material latents, ensuring What You See Is What You Get fidelity on visible regions and coherent completion on unobserved surfaces.

3. Bold header: Decoupling Visible-Material Estimation

By separating the tasks, the system trains an MLLM-based image editing model F2D to predict lighting-separated PBR maps in the reference view, shielding the 3D generator from abstract visual features and allowing it to focus on completing unobserved surfaces.

4. Bold header: Enhanced Material Estimation Fidelity

The use of a multimodal DiT backbone for F2D, trained with a latent flow-matching objective at the fixed maximum-noise endpoint σ = 1.0, enables the estimation of clean material latents in a single forward pass, which uses 2D image priors to disentangle illumination from intrinsic materials.

5. Bold header: Adaptive Query Strategy in Flow Transformer

The interleaved flow transformer employs a dynamic query schedule where query coordinates alternate across blocks: Jl = U for even l, and Jl = S for odd l, allowing the model to concentrate generative capacity on unobserved surfaces while maintaining global coherence through full-surface updates in odd layers.

6. Bold header: Robustness to Incomplete Observations

The system employs an encoding where it avoids assigning explicit material values to unobserved voxels, instead using the sparse coordinate architecture of Emat to omit unobserved voxels, which is shown to produce better results than zero-filled observations by avoiding a pronounced dark bias on unseen surfaces.