SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation

arXiv:2509.24980 · cs.CV · Submitted 2025-09-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation".

Jane: Here's the gist of the paper. The gist The authors propose SDPose, a pose estimation framework that repurposes a pre-trained Stable Diffusion U-Net as the backbone and predicts keypoints from its upsampling-block features.

Tom: Systematic Analysis of Latent Features The researchers systematically analyzed the latent features from different upsampling levels of the Stable Diffusion U-Net to identify which levels deliver the strongest robustness and cross-domain generalization for pose estimation.

Jane: They used t-SNE and silhouette scores on 1200 person crops to compare features from pre-trained Sapiens-1B and the SD-v2 U-Net.

Tom: Based on this analysis, they concluded that they select a robust subset F ⊂

F1, F2, F3: for subsequent feature aggregation and heatmap prediction.

Paper summary: Tom: So we're wrapping up on SDPose, which is this paper about using Stable Diffusion U-Net features for pose estimation in totally different visual styles and conditions.

Jane: Right, so they figured out that by looking at which parts of the U-Net’s latent space are most robust across different people and environments—that’s what they did.

Lu: They systematically analyzed those upsampling levels, t-SNE plots, and silhouette scores to pick a specific subset of features for their pose prediction backbone.

Meng: I'm curious about the practical side here. How does this feature selection actually translate to something we can deploy without needing a massive compute budget?

Lalam: The paper shows that SDPose uses those selected features, fuses them with a small aggregation module, and adds a reconstruction loss to keep the generative prior strong during training.

Tom: So it’s not just taking raw U-Net output; it’s building a new representation by carefully curating those latent features and adding that regularization loss.

Jane: And when they tested it on COCO-OOD, with style transfers and various corruptions, the results were quite strong for zero-shot performance.

Lalam: They achieved state-of-the-art results on HumanArt and COCO subsets while requiring shorter fine-tuning schedules compared to models like Sapiens.

Meng: That makes sense from a production standpoint if the training time is reduced and we’re still getting high performance without needing those enormous foundation models for every single task.

Lu: It does show that repurposing these diffusion priors can be effective for cross-domain tasks, which is a new angle for how we think about using pre-trained generative models.

Tom: So the main point of this paper is that you can leverage the latent structure of a powerful image model to build a more specialized and robust system for pose estimation across different visual domains.

Jane: It’s about making AI models more versatile, so they don't just work well on the exact data they were trained on.

Lalam: This zero-shot capability means you could potentially use this as a pose annotator for controllable generation pipelines in any style or lighting condition.

Tom: It opens up a new way to guide image generation using pose information that stays consistent even when the visual input changes drastically.

Jane: And we’ve got some interesting questions about how far this robustness really extends, and what happens when we push the limits of those cross-domain tests.

Conclusion: Tom: So we're wrapping up on SDPose, which is this paper called "SDPose: Exploiting Diffusion Priors for Out-of-Domain and Robust Pose Estimation."

Jane: Right. This paper looks at how to use those diffusion models, specifically the Stable Diffusion U-Net, not just for making pictures, but for figuring out human poses.

Lu: They figured out that by looking at which parts of the U-Net’s latent space are most reliable across different people and environments—that's what they did.

Meng: It seems like the core idea is selecting a specific subset of those latent features because they generalize better when you need to estimate poses in completely different visual styles or domains.

Lalam: The authors propose SDPose, which takes those chosen features, merges them with a small module to make a robust representation, and uses an auxiliary loss to keep the original generative knowledge strong while training for pose estimation.

Tom: So it’s about curating the U-Net features by systematically analyzing different upsampling levels to find the ones that give you the most stable results when things get messy.

Jane: And they tested this robustness on a benchmark called COCO-OOD, which includes style shifts and various visual corruptions like noise or blur.

Lu: Their analysis of t-SNE dimensions and silhouette scores across different feature blocks showed that certain levels performed significantly better in terms of both person identity consistency and style transfer capability.

Meng: For the folks who are deploying this, the paper shows that with data-efficient fine-tuning, SDPose gets performance numbers comparable to Sapiens models on COCO but uses a smaller parameter budget.

Lalam: They even show better whole body AP and AR scores on datasets like COCO-OOD Monet and Ukiyo-e compared to Sapiens models.

Tom: It really boils down to this paper showing you can leverage the latent structure of these big generative models to build a pose estimation system that stays consistent even when the visual input looks totally different from what the model was originally trained on.

Jane: That means we're moving toward AI that can give you reliable pose guidance for controllable generation pipelines, no matter the style or lighting condition of the image you feed it.

Lu: It builds on earlier work where diffusion priors were used for things like depth and surface normal estimation, but this applies that idea directly to cross-domain human pose estimation.

Meng: I do wonder if relying so heavily on these pre-trained models means we're just sharpening existing biases in their feature extraction instead of solving a fundamental representation problem.

Lalam: The authors are pretty sure about it because they use that reconstruction loss to actively preserve the generative quality while they train for the pose task.

Tom: So, to sum up this paper, SDPose finds which U-Net feature blocks are best for pose tasks, builds a fusion module on those blocks, and uses that reconstruction loss to maintain the prior while hitting state-of-the-art results on cross-domain benchmarks.

Jane: And what this means is we have a new way to get reliable poses from images that look very different from what the model was originally trained on, which is a big step for making AI models more versatile and useful in real applications.

Tom: We'll talk more next about how this zero-shot capability changes how artists use these models.

Shuang Liang, Jing He, Chuanmeizhi Wang, Lejun Liao, Guo Zhang, Yingcong Chen, Yuan Yuan

Rama Alpaca Technology Boston College Research Institute of Tsinghua University in Shenzhen HKUST The University of Hong Kong Research Institute of Tsinghua University in Shenzhen HKUST

cs.CV

Submitted: 2025-09-29

Updated: 2026-10-21

Comments: 23 pages, 11 figures, 12 tables

Code: https://github.com/MooreThreads/Moore-AnimateAnyone

Project page: https://t-s-liang.github.io/SDPose

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist The authors propose SDPose, a pose estimation framework that repurposes a pre-trained Stable Diffusion U-Net as the backbone and predicts keypoints from its upsampling-block features.

Key concepts

Stable Diffusion U-Net
This is the core neural network from Stable Diffusion, a powerful model used for image generation. The researchers use its internal layers to extract rich visual features that capture complex information about an input image, which they then repurpose for pose prediction tasks.
Latent Features Analysis
The authors systematically examined the feature maps produced by different upsampling levels of the U-Net. They used tools like t-SNE and silhouette scores to determine which specific layers provide the most stable and generalizable information for estimating human poses across various visual domains.
Cross-Domain Generalization
This refers to a model's ability to perform well on data it was not explicitly trained on, such as different artistic styles or images corrupted by noise. SDPose is designed to be robust because it leverages the pre-trained generative knowledge of the diffusion model to maintain accuracy when facing these unseen visual conditions.
Auxiliary Latent Reconstruction Loss
This is a training technique used in SDPose. It adds an extra loss function during training that forces the model's latent features to reconstruct the original input image. This helps regularize the training process, ensuring that the extracted features remain meaningful and preserve the generative prior learned by Stable Diffusion.

Terminology

Summary

The gist The authors propose SDPose, a pose estimation framework that repurposes a pre-trained Stable Diffusion U-Net as the backbone and predicts keypoints from its upsampling-block features.

Systematic Analysis of Latent Features

The researchers systematically analyzed the latent features from different upsampling levels of the Stable Diffusion U-Net to identify which levels deliver the strongest robustness and cross-domain generalization for pose estimation. They used t-SNE and silhouette scores on 1200 person crops to compare features from pre-trained Sapiens-1B and the SD-v2 U-Net. Based on this analysis, they concluded that they select a robust subset F ⊂ [F1, F2, F3] for subsequent feature aggregation and heatmap prediction. On this benchmark, SDPose achieves state-of-the-art results on HumanArt and COCO-OOD with fewer training epochs and a smaller parameter budget. On HumanArt, SDPose sets a new state-of-the-art with 71.8 AP / 74.6 AR, surpassing large-scale foundation baselines under the same COCO-only training. For the Wholebody variant, SDPose surpasses the parameter-matched Sapiens-1B by +0.1 AP and +0.2 AR on the COCO-WholeBody validation set and achieves a whole-body AP/AR of 72.8/79.4 on COCO-OOD Monet and surpasses Sapiens-2B on COCO-OOD Ukiyo-e with a whole-body AP/AR of 47.7/56.4 and demonstrates superior robustness on the COCO-OOD Corruption dataset compared to Sapiens-2B and surpasses Sapiens-2B in overall Whole AP (54.3 vs. 52.8)"> Finally, SDPose serves as a zero-shot pose annotator for controllable generation pipelines, providing stable pose guidance across diverse visual domains.

Downstream Applications

SDPose provides more precise and detailed skeletons than DWPose, enabling reliable pose transfer and high-quality image generation for artistic characters. Furthermore, SDPose enables reliable pose sequence transfer from animations to animations.

Conclusion

SDPose establishes new state-of-the-art performance on HumanArt and multiple COCO-OOD subsets while requiring shorter fine-tuning schedules and a smaller parameter scale compared to Sapiens models. The framework is demonstrated to be effective for zero-shot pose annotation in controllable generation pipelines.

Improvements for AI systems

  1. System can perform robust pose estimation on out-of-domain images, specifically stylized paintings and images of humanoid robots, achieving more accurate predictions than Sapiens and ViTPose++-H as shown in Figure 1.

  2. The system will achieve state-of-the-art performance on HumanArt and COCO-OOD benchmarks with shorter fine-tuning schedules and a smaller parameter scale compared to existing models, demonstrating superior cross-domain generalization.

  3. The model can serve as a zero-shot pose annotator for downstream controllable generation tasks, such as ControlNet based image synthesis and video generation, providing reliable and stable pose guidance across diverse visual domains.

  4. The feature aggregation module will fuse selected U-Net features from different upsampling blocks (F1, F2, F3) to combine complementary cues across U-Net levels, improving OOD behavior without sacrificing in-domain accuracy for both body and whole-body estimation.

  5. The auxiliary RGB reconstruction branch will be incorporated during fine-tuning to preserve the generative priors of the pre-trained backbone, acting as a regularizer that improves cross-domain generalization under style shifts.

Abstract

Pre-trained diffusion models provide rich latent features across U-Net levels and are emerging as powerful vision backbones. While prior works such as Marigold and Lotus repurpose diffusion priors for dense geometric perception tasks such as depth and surface normal estimation, their potential for cross-domain human pose estimation remains largely unexplored. Through a systematic analysis of latent features from different upsampling levels of the Stable Diffusion U-Net, we identify the levels that deliver the strongest robustness and cross-domain generalization for pose estimation. Building on these findings, we propose SDPose, which (i) extracts U-Net features from the selected upsampling blocks, (ii) fuses them with a lightweight feature aggregation module to form a robust representation, and (iii) jointly optimizes keypoint heatmap supervision with an auxiliary latent reconstruction loss to regularize training and preserve the pre-trained generative prior. To evaluate cross-domain generalization and robustness, we construct COCO-OOD, a COCO-based benchmark with four subsets: three style-transferred splits to assess domain shift and one corruption split (noise, weather, digital artifacts, and blur) to test robustness. With data-efficient fine-tuning, SDPose achieves performance comparable to Sapiens on COCO, and establishes the best zero-shot performance on cross-domain datasets.

Sources

Related papers