TAMF-VTON: Token-Wise Sparse Adaptation for Mask-Free Virtual Try-On

arXiv:2607.14807 · cs.CV · Submitted 2026-07-16 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "TAMF-VTON: Token-Wise Sparse Adaptation for Mask-Free Virtual Try-On".

Jane: Here's the gist of the paper. The gist: TAMF-VTON presents a texture-aware, mask-free framework that enables high-fidelity image synthesis under practical unconstrained conditions by formulating garment transfer as a task-specialized image editing problem Page 1.

Tom: How it works TAMF-VTON is a textureaware, mask-free framework that enables high-fidelity image synthesis under practical unconstrained conditions Page 1.

Jane: It requires no human parsing or inpainting masks at inference time and supports diverse garment styles, categories, and quantities Page 1.

Tom: The framework is achieved through a unified generative pipeline with three key components Page 1.

Title and authors: Tom: So we're looking at the paper called "TAMF-VTON: Token-Wise Sparse Adaptation for Mask-Free Virtual Try-On." It sounds like they’ve tackled a big problem in virtual try-on because it’s trying to get high fidelity synthesis without needing those annoying segmentation masks.

Jane: That's right, Tom. The title itself tells you the main goal: mask-free virtual try-on using token-wise sparse adaptation. Mask-free means no human parsing or preprocessed masks needed at inference time, which is a huge practical win for deploying this kind of tech in real use.

Lu: It’s interesting because it shifts the entire problem formulation. Instead of treating virtual try-on as a mask-guided inpainting task, they treat it as a task-specialized image editing problem rather than just filling in missing parts <ref:2607.14807#pg3>.

Meng: So they’re trying to make the process cleaner on the model side instead of relying on messy masks for every single try-on attempt, which makes sense from a production standpoint if it works consistently.

Lalam: I see it formulated as an image editing problem rather than inpainting, which suggests they are handling all those coupled constraints—like texture and identity preservation—simultaneously in one go <ref:2607.14807#pg3>.

The paper's summary: Tom: The summary of TAMF-VTON says they use a unified generative pipeline with three key components to achieve this mask-free, high-fidelity synthesis under unconstrained conditions <ref:2607.14807#pg1>.

Jane: That means there are three core mechanisms working together. One is this lightweight Mixture-of-Experts adaptation scheme that helps the model specialize without messing up its general editing capabilities <ref:2607.14807#pg4>.

Lu: And the second thing is a frequency-domain supervision mechanism, which explicitly optimizes high-frequency spectral consistency to keep those fine textures looking sharp <ref:2607.14807#pg5>.

Meng: The third component they use is this robust data curation pipeline employing an adaptive inpainting strategy to simulate the inverse VTON process for training data generation <ref:2607.14807#pg5>.

Lalam: That adaptive inpainting strategy is smart because it estimates the target garment extent based on both the reference garment semantics and the target person's pose, so the mask-free training pairs are dynamically adjusted for different silhouettes <ref:2607.14807#pg6>.

The paper's improvements: Tom: The authors point out several specific design details. First, they use the MoE adaptation module to handle spatially entangled objectives like garment deformation and texture preservation all at once <ref:2607.14807#pg4>.

Jane: They emphasize that different parts of the same image need different adaptation behaviors at the same time, which is a tough challenge for any model trying to learn these things together.

Lu: The frequency-domain supervision loss term Lf is defined as Lf = gamma(t) · Mf1 · Mf ⊙ (F(xˆ0) − F(xzero)), and they stress that gamma t emphasizes frequency supervision at low-noise stages <ref:2607.14807#pg5>.

Meng: And for the data, they compare their adaptive inpainting strategy, AIS, against non-adaptive inpainting strategies to show it’s much better because it models the relationship between the inpainting region and the reference garment <ref:2607.14807#pg11>.

Lalam: The core improvement is that this approach addresses coupled constraints—deformation, texture fidelity, identity consistency, and background preservation—without needing those error-prone segmentation masks <ref:2607.14807#pg8>.

Conclusion: Tom: So to wrap up TAMF-VTON, the authors claim it achieves superior perceptual quality and competitive quantitative performance compared to state-of-the-art methods <ref:2607.14807#pg5>.

Jane: The main implication is that by integrating this visual-context aware Mixture-of-Experts adaptation scheme, frequency domain supervision, and the adaptive data curation strategy, they solve the coupled challenges of texture fidelity and identity preservation without needing error prone segmentation masks <ref:2607.14807#pg8>.

Lu: It shows that this MoE-based adaptation and frequency-domain supervision combination is effective for high-fidelity mask-free virtual try-on, avoiding the common trade off between structural consistency and texture fidelity <ref:2607.14807#pg13>.

Meng: From a deployment view, they also showed it can run fast; inference takes under fifteen seconds on an NVIDIA RTX four thousand ninety with INT4 quantization, which makes it commercially viable for scalable deployment <ref:2607.14807#pg1>.

Lalam: This framework is designed to support flexible multi-garment composition with efficient inference, and they've shown strong robustness even in real world settings under complex conditions <ref:2607.14807#pg12>.

Tom: So the final word on TAMF-VTON is that it provides a more balanced performance across different metrics compared to strong baselines <ref:2607.14807#pg13>.

Jane: It really does make sense for people who only listen because it means you get high quality results without needing to painstakingly prepare every single image with masks <ref:2607.14807#pg2>.

Lu: We'll keep an eye on how they refine that MoE adaptation further, because handling spatially entangled objectives is definitely where the future of this type of fine-tuning lies.

Meng: I just wonder about the cost at scale, but if inference is under fifteen seconds on consumer hardware, that’s a major operational win for any production pipeline we're building.

Lalam: And I'm confident in what they present; they address the core issues of texture and composition without forcing a human to do tedious mask work.

Zhejiang University and Style3D Research

cs.CV

Submitted: 2026-07-16

Updated: 2026-10-20

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: TAMF-VTON presents a texture-aware, mask-free framework that enables high-fidelity image synthesis under practical unconstrained conditions by formulating garment transfer as a

Key concepts

Mixture-of-Experts (MoE) Adaptation: This scheme uses a token-wise router to dynamically select the most relevant experts within the model for each part of the image. It allows the model to learn task-specific try-on skills efficiently without corrupting its general ability to edit images, handling complex constraints like garment deformation and texture preservation simultaneously.
Frequency-Domain Supervision: This mechanism explicitly optimizes high-frequency spectral consistency. By focusing on high frequencies in the loss term, the framework ensures that fine textural details and material appearances are accurately preserved in the synthesized output, leading to much higher visual quality than standard methods.
Adaptive Inpainting Strategy: This pipeline simulates the inverse VTON process to generate mask-free training pairs. It estimates where a garment should be placed based on body keypoints and reference garment aspect ratios, allowing it to dynamically adjust masks for different styles and silhouettes, creating diverse training data.
Mask-Free Inference: The framework enables high-fidelity synthesis without requiring any human parsing or inpainting masks during the actual try-on process. This makes the system practical for real-world deployment in unconstrained settings where manual mask creation is infeasible.

Terminology

Adaptation — 2× in this paper · explained in 1 episode
Fidelity — 2× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Mixture-of-Experts (MoE) — 1× in this paper · explained in 10 episodes
Quantization — 1× in this paper · explained in 15 episodes
Segmentation Masks — 1× in this paper · explained in 2 episodes

Summary

The gist: TAMF-VTON presents a texture-aware, mask-free framework that enables high-fidelity image synthesis under practical unconstrained conditions by formulating garment transfer as a task-specialized image editing problem Page 1.

How it works

TAMF-VTON is a textureaware, mask-free framework that enables high-fidelity image synthesis under practical unconstrained conditions Page 1. It requires no human parsing or inpainting masks at inference time and supports diverse garment styles, categories, and quantities Page 1. The framework is achieved through a unified generative pipeline with three key components Page 1.

The three key components of the framework are:

A lightweight Mixture-of-Experts (MoE) adaptation scheme that enables efficient fine-tuning without compromising the base model’s general editing capabilities Page 1. This scheme uses a token-wise router to dynamically select the most relevant experts for each token, injecting try-on expertise while preserving the pretrained model’s general image editing capability Page 4.
A frequency-domain supervision mechanism that explicitly optimizes high-frequency spectral consistency to preserve high-fidelity textures Page 5. This mechanism is formulated as a loss term Lf, which enforces spectral consistency between the synthesized result and the ground-truth image by focusing on high-frequency components Page 5.
A robust data curation pipeline employing an adaptive inpainting strategy to simulate the inverse VTON process for high-quality training pair generation Page 5. This strategy estimates the inpainting region according to both reference garment semantics and target-person pose, allowing the synthesized inpainting mask to dynamically adjust to different garment lengths and silhouettes Page 6.

Key Design Details

The framework addresses coupled constraints by formulating virtual try-on as a task-specialized image editing problem Page 2. The MoE adaptation module is designed to handle spatially entangled objectives, such as garment deformation, texture preservation, identity consistency, and background preservation Page 4. Specifically, different spatial regions within the same image often require different adaptation behaviors simultaneously Page 4.

The frequency-domain supervision mechanism is crucial for texture fidelity Page 5. It focuses on high-frequency components, as these regions encode fine textural details Page 5. The loss Lf is formulated as Lf = γ(t) · Mf1 · Mf ⊙ (F(xˆ0) − F(x0)) Page 5, where γ(t) emphasizes frequency supervision at low-noise stages Page 5.

The adaptive inpainting strategy for data curation generates mask-free training tuples by first constructing a reference-agnostic base image using a mask-based VTON model Page 6. This strategy estimates the target garment extent according to body keypoints and the aspect ratio of the reference garment region, ensuring coverage of both the expected reference-garment footprint and residual textures from the original clothing Page 6.

Experimental Validation

Extensive experiments demonstrate that TAMF-VTON outperforms state-of-the-art methods in both quantitative metrics and perceptual quality Page 1. The model achieves inference in under 15 seconds per image on an NVIDIA RTX 4090 with INT4 quantization, making it commercially viable for scalable deployment Page 1.

The ablation studies confirm the effectiveness of the proposed components Page 6. The effect of MoE adaptation shows that the MoE variant produces more stable and realistic results across challenging cases compared to a single-LoRA baseline, which shows limited generalization ability Page 6. Furthermore, the frequency-domain supervision significantly improves the preservation of fine-grained fabric textures and material appearance when compared to models without it Page 6.

The adaptive inpainting strategy is validated by comparing it with non-adaptive inpainting strategies (NAIS) Page 11. The AIS explicitly models the relationship between the inpainting region and the reference garment, allowing for flexible transfer across diverse garment categories and styles Page 12.

Conclusion

TAMF-VTON achieves superior perceptual quality and competitive quantitative performance compared to state-of-the-art methods Page 5. By integrating a visual-context-aware Mixture-of-Experts adaptation scheme, frequency-domain supervision, and an adaptive data curation strategy, our method addresses the coupled challenges of garment deformation, texture fidelity, and identity preservation without relying on error-prone segmentation masks Page 8. This approach supports flexible multi-garment composition with efficient inference Page 1.

The paper concludes that TAMF-VTON demonstrates strong robustness and practical usability in real-world VTON settings under complex conditions Page 12. The method provides a more balanced performance across different metrics compared to strong baselines, avoiding the common trade-off between structural consistency and texture fidelity Page 13. This demonstrates the effectiveness of our MoE-based adaptation and frequency-domain supervision for high-fidelity mask-free virtual try-on Page 8. The framework is optimized for efficiency, achieving inference in under 15 seconds per image on consumer hardware Page 1.

The paper references numerous related works, including GAN-based VTON methods Page 2, mask-based VTON approaches like TryonDiffusion and StableVITON Page 2, and recent strong multimodal image editing models such as Qwen-Edit-2509 Page 13. The authors also compare TAMF-VTON against other strong baselines in the supplementary material, including Flux2-klein-9b and Nano Banana Pro Page 12. The paper presents quantitative results on VITON-HD and DressCode datasets under both paired and unpaired settings Page 5. The authors also present qualitative comparisons in challenging real-world scenarios, including subject-to-subject transfer and extreme lighting conditions Page 15. The paper references the use of OpenPose for extracting normalized 2D body keypoints Page 6. The paper references the use of SAM2 segmentations to protect hands and feet during inpainting Page 6. The paper references the use of YOLOv10 for detecting garment regions Page 6. The paper references the use of VITON-HD and DressCode datasets for training and evaluation Page 5. The paper references the use of Flow matching for formulating the generative process Page 3. The paper references the use of Lora and IP-Adapter as prior adaptation techniques Page 2.

Improvements for AI systems

Mask-Free Inference Capability Enables Practical Deployment: TAMF-VTON requires no segmentation masks, human parsing, or auxiliary control signals during inference. This allows for deployment in practical unconstrained conditions without the error-prone preprocessing of mask-based methods which rely on segmentation masks.
Lightweight MoE Adaptation for Task Specialization: The framework introduces a lightweight Mixture-of-Experts (MoE) adaptation scheme that injects task-specific try-on expertise into the diffusion model while preserving the pretrained model’s general image editing capability. This provides task-specific capacity for virtual try-on without overwriting the pretrained diffusion model.
High Texture Fidelity via Spectral Consistency: The method utilizes a frequency-domain supervision mechanism with a loss term, formulated as L = LFM + λLf, to explicitly encourage spectral consistency between the synthesized result and the ground-truth image, thereby improving high-frequency texture consistency.
Flexible Multi-Garment Support: TAMF-VTON naturally supports both single-garment transfer and flexible multi-garment composition through token concatenation, enabling the simultaneous transfer of multiple items while preserving body structure and intricate texture details.
Adaptive Data Curation for Cross-Category Training: The adaptive inpainting strategy simulates the inverse VTON process to generate high-quality training pairs, specifically designed to create mask-free training triplets that are not constrained by style, category, or length similarity between the reference garments and the garments originally worn by the target person.
Enhanced Robustness in Challenging Scenarios: The system is robust under complex conditions, as demonstrated by its ability to handle subject-to-subject transfer, challenging body-shape variations, and extreme lighting conditions while maintaining strong background preservation.