TED-VITON: Transformer-Empowered Diffusion Models for Virtual Try-On (2024)

3.1 Background on Controlling Diffusion Models

Stable Diffusion (SD) 3 Model. The SD 3 model [8] represents a significant breakthrough as the first diffusion model to utilize a Transformer-based architecture. Building upon Latent Diffusion Models (LDMs) [36], SD3 introduces the rectified flow approach [25], which connects data points in the latent space via straight linear paths, replacing the traditional curved trajectories. This straight-line trajectory minimizes noise accumulation and allows for efficient, high-quality image synthesis.

In SD3, the input image $x$ is encoded into a latent representation $z_{0}=E(x)$ by a pre-trained encoder $E$. The rectified flow formulation defines a forward diffusion process with a variance schedule $\beta_{t}$ expressed as follows:

where $t\in\{1,\dots,T\}$ indicates diffusion steps, $\alpha_{t}:=1-\beta_{t}$, and $\bar{\alpha}_{t}:=\prod_{s=1}^{t}\alpha_{s}$. SD3 leverages the Conditional Flow Matching (CFM) loss to guide rectified flow during training:

where $u_{t}(z|\epsilon)$ denotes the rectified vector field for direct, linear alignment. Unlike prior diffusion models, SD3 employs a Transformer backbone (Multimodal DiT) that facilitates bidirectional information flow between text and image tokens. This multimodal structure enhances text comprehension and visual quality, making SD3 highly suitable for text-guided image generation with improved fidelity and detail retention.

ControlNet for Conditional Image Generation. ControlNet [47] extends diffusion models by enabling conditional image generation with additional guidance inputs, such as edge maps, segmentation masks, or pose annotations. ControlNet operates by branching from the base model’s intermediate features $F$. The conditional input $C$ is processed with learnable weights $W_{c}$, resulting in conditioned features $F_{\text{ctrl}}=\text{ControlNet}(C;W_{c})$. These conditioned features are merged back into the main pipeline as $F_{\text{combined}}=F+\lambda F_{\text{ctrl}}$, where $\lambda$ regulates the influence of the conditional input. During training, ControlNet minimizes a composite loss:

where $\mathcal{L}_{\text{diff}}$ is the base diffusion model’s loss, $\mathcal{L}_{\text{cond}}$ ensures alignment with the guidance input, and $\gamma$ balances their contributions. This framework enables precise control over image generation, making ControlNet highly effective for tasks requiring fine-grained customization.

3.2 TED-VITON

Figure 2 (a) illustrates the TED-VITON framework, comprising DiT-GarmentNet, the Garment Semantic (GS) Adapter and DiT-TryOnNet. The following section provides a comprehensive description of each module and the training procedure.

DiT-GarmentNet. DiT-GarmentNet is designed to extract fine-grained garment features, including textures, patterns, fabric structures, logos, and other subtle design elements essential for realistic VTO results. By preserving the garment’s true visual characteristics, this module ensures high fidelity, particularly in applications requiring precise appearance rendering.

DiT-GarmentNet processes the latent representation of the garment image $\mathcal{E}(X_{g})$, extracted via a pre-trained VAE encoder $\mathcal{E}$, along with the conditioned text prompt $\tau_{\theta}(D)$ generated by a multi-modal text encoders. These representations flow through multiple transformer layers, refining and retaining intricate garment details. The transformer architecture, inspired by Esser etal. [8], captures long-range dependencies, ensuring consistent textures, accurate logo placement.

In this way, DiT-GarmentNet ensures high visual fidelity by combining garment-specific details with broader model context, enabling the VTO system to accurately render complex designs on various body shapes and poses.

Garment Semantic Adapter (GS-Adapter). The GS-Adapter [45] is a key module that enhances generalization, making the model less sensitive to variations in body poses, garment deformations, and conditions like lighting or camera angles. By focusing on low-frequency features, it captures essential garment attributes, enabling consistent performance across diverse scenarios.

Unlike DiT-GarmentNet, which extracts high-frequency details like textures and logos, the GS-Adapter uses the DINOv2 encoder [32] to distill semantic garment information, including structure, style, and material. These high-order semantics, $H_{\text{semantic}}$, encapsulate broader contextual attributes while maintaining adaptability.

The GS-Adapter employs a decoupled cross-attention mechanism to independently process joint and image embeddings. Let $\mathbf{Q}\in\mathbb{R}^{N\times d}$ represent the query matrix, and $\mathbf{K}_{j},\mathbf{V}_{j}$ and $\mathbf{K}_{i},\mathbf{V}_{i}$ denote key-value pairs for joint and image embeddings, respectively. The combined output is:

where $\lambda$ balances image and joint feature contributions. This design allows the GS-Adapter to generalize effectively across diverse poses, complex garments, and varying environmental conditions, enhancing model robustness and ensuring realistic outputs.

DiT-TryOnNet. DiT-TryOnNet builds upon the DiT architecture, leveraging its powerful Transformer-based diffusion capabilities within the latent space of a pre-trained VAE. By integrating DiT, our model benefits from the scalability and long-range dependency modeling of Transformers, enabling precise alignment and realistic rendering in virtual try-on scenarios. For DiT-TryOnNet, we construct a combined input $\zeta=[\mathcal{E}(X_{\text{model}});m;\mathcal{E}(X_{\text{mask}});\mathcal{E$ to provide a comprehensive context. This input consists of: the person’s latent image representation $\mathcal{E}(X_{\text{model}})$ as the primary structural guide; a dynamically resized mask $m$ to isolate the garment area and focus the model’s attention; the masked person’s image $X_{\text{mask}}=(1-m)\odot X_{\text{model}}$ for garment reconstruction; and the DensePose embedding $\mathcal{E}(X_{\text{pose}})$ to align with the person’s pose.

Within the MM-DiT-Block (Fig.2(b)), fine-grained garment details $F_{\text{garment}}^{i}$ extracted from the $i$-th transformer layer of DiT-GarmentNet, merge with the feature representation $F_{\text{tryon}}^{i}$ from the corresponding $i$-th layer DiT-TryOnNet to form $F_{\text{image}}^{i}$, which serves as the primary input for attention processing. Descriptive text embeddings $\tau_{\theta}(D)$, generated by multimodal text encoders, are concatenated with $F_{\text{image}}^{i}$ within the query, key, and value components of the joint attention mechanism (i.e., $Q_{\text{joint}}=\text{Concat}(Q_{\text{image}}^{i},Q_{\tau_{\theta}(D)})$, $K_{\text{joint}}=\text{Concat}(K_{\text{image}}^{i},K_{\tau_{\theta}(D)})$, $V_{\text{joint}}=\text{Concat}(V_{\text{image}}^{i},V_{\tau_{\theta}(D)})$). This results in a hidden state $H_{\text{joint}}^{i}$ that unifies visual and textual modalities. Subsequently, this hidden state is further enriched by incorporating high-order semantic features, $H_{\text{semantic}}$ provided by the GS-Adapter, as described in Eq. 5.

To produce the final VTO output $\hat{X}$, DiT-TryOnNet leverages the combined input $\zeta$ and garment description embedding $\tau_{\theta}(D)$:

Prior Preservation for Text Generation. To retain the model’s ability to generate accurate and clear text, such as logos and labels, we introduce a prior preservation mechanism inspired by DreamBooth [37]. This mechanism incorporates a text preservation loss to ensure text clarity and fidelity, preventing the model from losing this capability while fine-tuning for VTO tasks. As the final component of our framework, prior preservation complements the GS-Adapter and DiT-TryOnNet. Together, they form a comprehensive training objective, achieving a balance between high-fidelity garment rendering and robust text generation for realistic VTO outputs.

As shown in Fig.2(a), the total loss function combines two main components: (1) the CFM loss $\mathcal{L}_{\text{CFM}}$ defined in Eq.2, which ensures high-quality VTO outputs by aligning generated images with the desired garment and pose, and (2) the text preservation loss $\mathcal{L}_{\text{pres}}$, which maintains clarity in text details. The CFM loss guides the model in generating the VTO result $\hat{X}$ leveraging DiT-GarmentNet for detail retention and DiT-TryOnNet for fit adjustments based on pose and body type. The text preservation loss $\mathcal{L}_{\text{pres}}$ is computed as $\mathcal{L}_{\text{pres}}=\text{MSE}(\hat{X},\hat{X}^{\prime})$, where $\hat{X}^{\prime}$ is the baseline latent representation from the original model, helping to retain text fidelity in the fine-tuned output. The final loss function is given by:

where $\lambda_{\text{pres}}$ controls the balance between VTO adaptation and text retention. This approach enables high-quality garment realism while preserving essential text rendering for realistic try-on images.

GPT-4o Generated Garment Descriptions. Our approach uses GPT-4o to generate detailed garment descriptions that capture both basic and nuanced features. These descriptions provide rich semantic context, enhancing the model’s ability to faithfully represent garment details. For DiT-GarmentNet, descriptions help preserve intricate details like texture and logos. Meanwhile, for DiT-TryOnNet, the text prompt is tailored to emphasize how the garment appears when worn, focusing on fit and interaction with the body. This adjustment improves realism in the generated images. This dual-conditioning approach enables more accurate garment representation, as shown in Fig.2(a).

4.1 Experiment Setup

Baselines. We evaluate our method against both GAN-based and diffusion-based VTO approaches. The GAN-based baselines include HR-VITON [21] and SD-VTON [38]. Both methods employ a separate warping module to fit the garment onto the target person, followed by GAN-based generation with the fitted garment as input. Among the diffusion-based methods, we compare with LaDI-VTON [30], DCI-VTON [11], StableVITON [19], and IDM-VTON [5]. All of these models leverage pretrained SD models, though with different conditioning techniques. LaDI-VTON and DCI-VTON incorporate distinct warping modules for garment conditioning, while StableVITON directly uses the SD1.4 encoder for conditioning. IDM-VTON, by contrast, utilizes the SDXL inpainting model checkpoints from official repositories. Our approach similarly builds on SD3 original checkpoints from official sources. For a fair comparison, we generate images at a resolution of $1024\times 768$ when available; otherwise, we generate images at $512\times 384$ and upscale them to $1024\times 768$ using interpolation or super-resolution techniques [41], reporting the highest-quality results achieved.

Evaluation datasets. We evaluate the effectiveness of TED-VITON on two widely-used VTO datasets, VITON-HD [4] and DressCode [29]. The VITON-HD dataset consists of 13,679 pairs of frontal-view images of women and corresponding upper garments. Following the standard dataset practices of previous works [30, 11, 19, 5, 39], we divide VITON-HD into a training set of 11,647 pairs and a test set of 2,032 pairs. The DressCode dataset contains 15,366 image pairs focused specifically on upper-body garments. Consistent with the original dataset splits, we use 1,800 upper-body image pairs from DressCode as the test set. All experiments on both VITON-HD and DressCode are conducted at a resolution of $1024\times 768$.

Evaluation metrics. We evaluate TED-VITON in both paired and unpaired settings, following established practices in VTO literature. In the paired setting, the input garment matches the one originally shown in the person image. To assess performance, we use three key metrics: Structural Similarity Index (SSIM) [42], Learned Perceptual Image Patch Similarity (LPIPS) [48] and the CLIP image similarity score (CLIP-I) [14] to measure similarity between the generated image and the ground truth. Additionally, in the unpaired setting, where the garment in the person image is replaced with a different one and no ground truth is available, we assess TED-VITON’s performance in terms of image quality and realism using Fréchet Inception Distance (FID) [15] and Kernel Inception Distance (KID) [18] scores.

4.2 Qualitative Results

Fig.3 provides a qualitative comparison of VTO models alongside the input person image and selected garments. TED-VITON stands out as the only model capable of accurately reproducing text details on garments, such as the large “1969” and “Wrangler” logos, as well as finer text like “Vans”. In terms of color and texture fidelity, TED-VITON precisely aligns the four colors in “1969” across the text rows, maintaining the original garment’s design. Unlike other models, which often exhibit text distortion or color misalignment, TED-VITON preserves text clarity and color accuracy. This is achieved through the integration of a Text Preservation Loss and enhanced prompt conditioning, which together ensure that fine-grained text and color details are retained in the generated VTO images.

4.3 Quantitative Results

VITON-HD. We evaluate TED-VITON on the VITON-HD dataset and compare it with SOTA VTO methods, including GAN-based approaches (HR-VITON [21] and SD-VITON [38]) and diffusion-based methods (LaDI-VTON [30], DCI-VTON [11], StableVITON [19], and IDM-VTON [5]). Table 1 presents the quantitative results, where TED-VITON achieves top scores in LPIPS, CLIP-I, FID, and KID, indicating superior perceptual quality and realism. It ranks second in SSIM, highlighting its strong structural similarity preservation and alignment with perceptual semantics.

DressCode Upper-body. To evaluate TED-VITON’s generalization across diverse garment styles, we test it on the DressCode upper-body dataset. As shown in Table 1, TED-VITON outperforms other models across most metrics, achieving top scores in LPIPS, SSIM, CLIP-I and FID scores, which indicate strong alignment with the perceptual features of the in-shop garment. TED-VITON outperforms diffusion-based models like IDM-VTON and StableVITON by consistently capturing finer patterns and more accurate garment textures. In contrast, GAN-based methods struggle with complex patterns, resulting in lower-quality outputs on this dataset.

4.4 User Study

To complement objective metrics, we conducted a user study to evaluate the visual appeal of our model. The study used 10 image pairs from the VITON-HD dataset, divided into two groups: 5 pairs focused on text and logo clarity, and 5 pairs evaluated pattern and texture fidelity. As shown in Fig.5, with 50 valid responses, TED-VITON was the preferred model, demonstrating strong user preference for text clarity and pattern accuracy.

4.5 Ablation Study

Effect of key components of TED-VITON. To analyze the contributions of each key component in TED-VITON, we conduct an ablation study by systematically removing individual components, such as DINOv2, the GS-Adapter, DiT-GarmentNet, or Text Preservation Loss, and evaluate their impact on the results. In Fig.4(a), replacing DINOv2 with a standard CLIP encoder results in blurry and distorted text. This highlights DINOv2’s role, in conjunction with the GS-Adapter, in enhancing text clarity and garment alignment by capturing fine semantic and garment details. Fig.4(b) demonstrates the impact of removing the GS-Adapter, leading to misaligned garment features and reinforcing its importance for detailed garment representation. As shown in Fig.4(c), removing DiT-GarmentNet compromises fine garment details, like textures and logo placement, indicating its role in preserving intricate design elements. In Fig.4(d), without Text Preservation Loss, text appears slightly distorted, emphasizing this loss function’s role in maintaining text fidelity. As shown in Fig.4(e), the full model achieves optimal performance by incorporating all components, accurately capturing both structural and stylistic details. Quantitative evaluation in Table2 further supports these observations.

Effect of using GPT-generated captions. We conducted an ablation study to evaluate the effect of detailed GPT-generated captions on TED-VITON’s performance. As shown in Fig.6, utilizing GPT-generated captions significantly improves the model’s ability to render multi-line text and garment details like color accuracy and texture. Using a brief description, the model accurately renders the first line, “SUN”, but fails to capture “SAND” and “SURF” due to insufficient contextual guidance. Using a detailed caption provides the necessary guidance for accurately rendering all lines and maintaining consistent color and texture. Such quantitative improvements across all metrics are demonstrated in Table3.