From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches

arXiv:2610.21865 · cs.GR, cs.CV, cs.HC · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "From Sketch to Scene".

Jane: The gist The system transforms a rough garment sketch,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on to the title and authors of this paper, "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". It’s clear they are focusing on taking a very simple input—a sketch with just a star for highlights and a moon for shadows—and turning it into something much more complex than what previous systems could do.

Jane: That title really tells you the main selling point, which is that it’s not just about getting a three dee model; it’s specifically about reconstruction and making sure the lighting is relightable <ref:2610.21865#pg1>. It moves beyond just generating geometry to defining how that geometry interacts with light.

Lu: They are trying to show that sketching can be a more direct way to capture spatial structure and intent, unlike text prompts which current systems rely on heavily >

Meng: So they’re arguing that the sketch captures the designer's specific intent for highlights and shadows directly, whereas text prompts are more abstract >

Tom: Right. The authors are Dogan, Ba¸saran, Sezgin, and Montazeri from Koç University and Manchester University. They cite their work on eliciting that two-symbol notation from art students as a key part of their methodology >

Jane: And they emphasize that they conducted an elicitation study with art students to derive the star for a highlight and the moon for a shadow, which shows this isn't just some arbitrary choice but something derived from real designer input >

Lu: It’s important because it grounds the notation in how actual designers think about visual ideas, moving it away from just being an abstract concept >

Meng: From what I see, they are setting up a solid foundation by defining exactly what those symbols mean and how they should be placed on the drawing itself >

Tom: So, if we look at this as a whole, the authors are building a pipeline that starts with these two simple marks and uses them to control everything from the final light direction to the surface displacement >

Jane: It’s about showing how those two symbols can be used as powerful controls instead of just being decorative elements on a drawing >

Lu: They are moving away from just having the shading baked into a texture, which is what they call "a nuisance" in their related work because you can't move the illumination afterward >

Meng: So this paper is proposing a system that treats those marks as an instruction set for both geometry and lighting simultaneously >

Tom: Exactly. The authors are proposing that instead of treating shading as something to be removed or baked in, they treat it as a direct input for generating the final scene structure >

Jane: So the big implication here is that we have a system where the light isn't fixed in the texture and can be moved and edited after generation >

Lu: It opens up possibilities because you’re not locked into one lighting scenario when you generate it; you’re getting a starting point that's inherently editable >

Meng: So this paper is showing how to build a system where the designer has control over the light in a more fundamental way than just picking a color map >

The paper's summary: Tom: Now let’s look at the actual summary of "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". It boils down to them conducting an elicitation study with art students to get their notation, which is the star for highlight and moon for shadow >

Jane: So they also detail how they predict that scene's directional light from those marks, which is a unit vector, and then they predict a displacement field from the marks to make sure the moon-marked regions fall into shadow under that predicted light >

Lu: And they combine these into a pipeline that turns a marked sketch and a texture prompt into this relightable three dee scene > <ref:2610.21865#pg1>

Meng: They also talk about how they synthesize the albedo separately, making sure the base color doesn't have illumination baked in so you can apply the predicted irradiance later >

Tom: So they’re essentially giving us a complete system that takes a sketch and a texture prompt and outputs geometry, material, lighting, and displacement information >

Jane: It’s a comprehensive pipeline where every part—the lighting prediction, the surface bending—is driven by those initial marks on the sketch >

Lu: They are showing how to use those marks as the input to drive both the light direction and the geometry simultaneously >

Meng: They are moving beyond just getting a single color map and adding this explicit directional light component, which is what makes it different from other texture-only stages >

Tom: So this paper’s main summary is that they create a system where you get an output that includes an explicit directional source ready to be relit >

Jane: It’s a complete scene with geometry, material properties, and the predicted lighting direction all bundled together for easy editing later >

The paper's improvements: Tom: Let’s look at the specific improvements they suggest in "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". They point out that a major improvement is treating shading marks as a first-class input, which we discussed >

Jane: And they also highlight how predicting the directional light from those marks allows for an editable environment, meaning the light is no longer fixed in the texture >

Lu: Another key thing they show is predicting a displacement field separately to handle shadows explicitly so that those moon-marked regions fall into shadow under that predicted light instead of being painted dark in the texture >

Meng: And they also mention how they combine these into a pipeline that turns the marked sketch and texture prompt into this fully specified glTF scene >

Tom: They are showing how to combine these components into one cohesive pipeline that handles the albedo synthesis, lighting prediction, and displacement field generation all at once >

Jane: And they also highlight how they separate the albedo synthesis so you can apply the predicted irradiance afterwards instead of baking it into the color >

Lu: So they are showing how to disentangling those parts is crucial because you can’t have illumination left in the texture if you want to move it later >

Meng: It really shows that separating them up makes sense for production because it prevents getting stuck with baked-in lighting issues >

Tom: The results from DeepFashionthree dee show that the light direction was recovered to twenty-two point nine degrees mean and forty-seven point five degrees median angular error, which is pretty concrete evidence of the system working on real data >

Jane: And they also showed that darkening at the marks showed that seventy-nine percent of the moon marks are darker than before, against forty-four percent of equally sized discs placed away from any mark on the same garments >

Conclusion: Tom: So to wrap up, this paper "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches" is about using a marked sketch input to get a relightable garment scene with an explicit directional light and geometry-driven shadows that you can edit later >

Jane: The big takeaway is that designers can specify lighting intent directly on the sketch, which changes how we think about generating three dee assets from sketches > <ref:2610.21865#pg1>

Lu: It validates the idea that these simple symbols are powerful enough to drive complex features like light and surface geometry simultaneously without needing a lot of extra training for every single mark >

Meng: For deployment, it means we can build systems that respect artistic intent regarding light and shadow in a more fundamental way than just relying on texture maps >

Lalam: I just want to say that the paper shows how this pipeline works by turning a marked sketch and a texture prompt into an output with the predicted light source as an explicit directional source alongside the textured mesh and the displacement field for accurate shadowing >

Tom: So it’s a system that takes those simple marks and gives you something fully specified, which is definitely worth looking at for anyone working in three dee generation or asset creation > <ref:2610.21865#pg1>

Jane: We’ve covered how this new approach lets designers specify lighting intent directly on the sketch rather than relying on baked-in textures >

Lu: We’ve seen how the paper validates that these simple symbols are capable of driving complex features like light and surface geometry without needing a lot of extra training for every single mark >

Meng: And for production, it means we can build systems that respect artistic intent regarding light and shadow in a more fundamental way than just relying on texture maps >

Lalam: So the "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches" paper is about getting an output with the predicted light source as an explicit directional source alongside the textured mesh and the displacement field for accurate shadowing >

Ahmet NeŞirvan Dogan, Eren Acar Başaran, Metin Sezgin, Zahra Montazeri

Koş University · University of Manchester

cs.GR, cs.CV, cs.HC

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: 11 pages, 5 figures. Ahmet Neçirvan Doğan and Eren Acar Başaran contributed equally

Code: https://github.com/KhronosGroup/glTF

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist The system transforms a rough garment sketch, marked with sparse annotations—a star for a highlight and a moon for a shadow—and a short text prompt describing the texture, into a

Key concepts

Shading Marks (Star/Moon): These are specific symbols placed on a garment sketch: a star indicates where the highlight should be, and a moon indicates where shadows will fall. The system treats these marks as direct instructions for determining the scene's lighting direction and surface geometry.
Displacement Field: This is a predicted field that bends the 3D mesh to create realistic folds and shadows. Instead of just coloring areas dark, this field physically pushes the surface points into shadow based on the predicted light direction, ensuring accurate shading.
Albedo Synthesis: This component generates the base color of the garment without applying any scene lighting first. By separating color synthesis from illumination, it ensures that when the predicted light is applied later, it correctly interacts with the material's inherent surface properties.

Summary

The gist The system transforms a rough garment sketch, marked with sparse annotations—a star for a highlight and a moon for a shadow—and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation.

How it works

The proposed pipeline treats shading marks as a first-class input to separate it from existing sketch-to-3D pipelines. The system reads the two symbols elicited from art students: a star where a highlight should fall and a moon where the fabric folds into shadow. These marks are placed directly on the drawing itself, and the pipeline predicts the scene’s directional light from these marks.

The system also predicts a displacement field separately from the light prediction. This displacement field bends the surface so that the moon-marked regions fall into shadow under that predicted light, rather than being painted dark in the texture.

Key Components and Predictions

The pipeline involves several stages to achieve this transformation. Specifically, the U-Net reads the marked sketch and the prompt to predict an irradiance map and a signed field describing folds. A light CNN then reads the U-Net’s bottleneck for the light direction, which is returned as a unit vector for the scene’s directional light.

The second output of the U-Net is a predicted field, which is a single signed value at every point of the garment from a 1 × 1 convolution on the decoder. This field, combined with crests placed using the predicted light, forms the displacement field that drives the mesh.

Albedo Synthesis and Composition

Surface color is produced by FLUX with a ControlNet that conditions it on the clean sketch and the text prompt. The albedo branch is trained to produce base color without the scene’s illumination, so that the predicted irradiance can be applied afterwards rather than being baked into the color.

The final composition combines the predicted irradiance E and the albedo ρ using a formula C = ρ (E +a) +s(1−r) f(ρ). This composite includes an ambient term 'a' that keeps regions the light does not reach at the color of the albedo instead of black, and a second term that adds a highlight based on the Blinn–Phong exponent derived from roughness r.

Displacement Mechanism

The goal of the displacement is to put each moon-marked region into shadow under the predicted light. The system uses a weighted request, where weight wi = λ (di/maxd) γ, to apply soft positional constraints on camera-facing vertices. An as-rigid-as-possible solve computes the final mesh by following these requests while preserving the surrounding surface.

To secure that side of a moon mark, one bump per mark is added alongside the predicted field, called the crest, which is raised so that the mark lies on its shadowed side. The crest’s size is set against that garment’s own predicted field through the 95th percentile of its magnitude.

Results and Performance

The system was evaluated on DeepFashion3D, where the light direction was recovered to 22.9◦ mean and 47.5◦ median angular error. Darkening at the marks showed that 79% of the moon marks are darker than before, against 44% of equally sized discs placed away from any mark on the same garments. On renders where no fold existed, displacement darkens 86% of these moon marks compared to 45% of control discs on the same garments.

The system successfully combines a marked sketch and a text prompt into a relightable garment scene with an explicit directional light. The final displaced mesh is exported as a glTF scene with the painted base-color texture, a single roughness value selected from the prompt, and the predicted directional light as an explicit directional source.

The system has limitations regarding how far a mark can be driven and artifacts that displacement can leave. The crest beside each mark takes its height from the predicted field and its side from the predicted light, but its footprint and its distance from the mark are fixed by a rule. The next step is predicting those parameters.

The system has limitations in how far a mark can be driven, in how the crest is placed, in the artifacts the displacement can leave, in the scene and material model, and in what we have evaluated. The coverage of professional designers remains an open question. The quantitative evaluation uses synthetic sketches rendered from DeepFashion3D, with marks placed by rule or by us rather than drawn by hand. A hand-drawn sketch, such as the one in Figure 1, appears only in qualitative results. The study is large enough to establish the star as the highlight symbol and the difference in hatching between the two categories, but not the moon’s lead over its runner-up. The two symbols are the most frequent proposals in an elicitation study with art students. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size. Both symbols we use are the modal proposals. The moon’s lead over its runner-up, the circle (6 against 4), is too small to be reliable at this sample size.

Improvements for AI systems

No current sketch-to-3D pipelines allow designers to specify lighting intent; this system transforms a marked sketch into a textured 3D mesh with an explicit directional light where the light is an editable part of the scene rather than fixed in the texture.
The system predicts the scene’s directional light from shading marks, allowing for a relightable environment, as described by the claim: We predict the scene’s directional light from these marks, so the light is an editable part of the scene rather than fixed in the texture.
The system predicts a displacement field to handle shadows explicitly: separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
A designer can specify shading directly on the sketch using symbols derived from elicitation studies: We propose treating shading marks as a first-class input, where a star for a highlight and a moon for a shadow are used to define lighting and surface geometry.
The system synthesizes appearance without baking illumination into the texture: The albedo synthesis branch is trained to produce base color without the scene’s illumination, so that the predicted irradiance can be applied afterwards rather than being baked into the color.
The final output is a fully specified glTF scene: The result is exported with the predicted light (Sec. 4.6), providing an explicit directional source alongside a textured mesh and the displacement field for accurate shadowing.

Sources

Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material