DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

arXiv:2610.12572 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models".

Jane: The gist The introduction focuses on online reinforcement learning for diffusion models, specifically addressing credit-assignment problems in reverse-process and forward-process methods,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to recap, DiffusionART is a new method that uses reward aggregation to fix the instability in NFT training and adds a second update for drift adaptive restoration without needing EMA tuning or a differentiable reward model. Jane Right, so it’s about taking those messy black-box rewards and smoothing them out by looking at the whole distribution across all prompts before applying them back into the reverse process states.

Lu: The authors of this paper "DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models" put the title there for a reason because they’re claiming efficiency and stability at scale.

Lu: The implication here is that we can move past the constant need to re-tune training setups every time we want to improve a model for a new generation task.

Meng: This approach could become a standard way to fine-tune diffusion models online, offering a more scalable training pipeline than what we see now with methods like Flow-GRPO.

Lalam: If this is robust enough for production traffic, it drastically cuts down on the compute time and the complexity of maintaining those reinforcement learning loops.

Conclusion: Tom: So to wrap up, DiffusionART is a new method that fixes the instability in NFT training by aggregating rewards first, and it adds a second update for drift adaptive restoration without needing EMA tuning or a differentiable reward model.

Jane: Right, so we’re talking about taking those messy black-box rewards and smoothing them out by looking at the whole distribution across all prompts before applying it back into the reverse process states.

Lu: The authors of this paper put the title DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models there for a reason—they are claiming efficiency and stability at scale.

Tom: They’re essentially showing that you can get high reward performance without relying on complex, task-specific tuning or fragile EMA schedules.

Meng: For someone shipping models to production, the implication is that we don't have to spend months painstakingly tuning an EMA schedule just to get stable training results for a new generation task.

Lalam: It suggests that reward guidance can be made more robust by looking at the overall distribution of rewards across prompts before applying it back into the reverse process states.

Jane: It means that for someone listening who only cares about the output, this paper suggests we can get better quality generation faster and more reliably without needing constant manual intervention on the training process itself.

Lu: The authors provide a clear path forward by showing how to stabilize diffusion reinforcement learning through reward aggregation and targeted restoration updates.

Tom: It’s about making these powerful models easier to train consistently across different tasks.

Jane: It moves the focus away from just tweaking hyperparameters like EMA schedules, towards structuring the reward signal itself for better long-term stability.

Lu: The implication is that this approach could become a standard way to fine-tune diffusion models online, offering a more scalable training pipeline than what we see now with methods like Flow-GRPO.

Meng: If this is robust enough for production traffic, it drastically cuts down on the compute time and the complexity of maintaining those reinforcement learning loops.

Lalam: I’m quite sure that DiffusionART’s use of pairwise reward models and its stability across image and video generation makes it a very solid direction for further research.

Zhenglin Wan, Jiayu Chen, †Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao

HPC-AI Lab, National University of Singapore 2Kling AI 3HKUST

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Project page: https://vanzll.github.io/diffusionart-project/ ; Code: https://github.com/vanzll/DiffusionART

Code: https://github.com/vanzll/DiffusionART

Project page: https://vanzll.github.io/diffusionart-project/ABSTRACT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist The introduction focuses on online reinforcement learning for diffusion models, specifically addressing credit-assignment problems in reverse-process and forward-process methods, and

Key concepts

Forward--Reverse Consistency Force: This is an inherent instability in NFT-style training where positive and negative reward branches create opposing forces on the model's velocity. Joint training only partially cancels this force, leading to erratic behavior. DiffusionART removes this by aggregating rewards first, ensuring a unified guidance direction for reverse-process states.
Aggregation-First Guidance: DiffusionART calculates a shared reward-improving direction G(c) by aggregating samples and rewards from all prompts for a given prompt. This is done by normalizing the covariance of rewards and endpoint means across all sampled endpoints. It provides a consistent, global signal to guide the reverse-process states without relying on task-specific sampling.
Two-Update Restoration: Instead of relying on Exponential Moving Average (EMA) for stabilization, DiffusionART uses a second MSE update with a trust-region mask. This update adds a restoration force proportional to the state's drift, effectively correcting deviations from the target velocity without requiring EMA tuning or per-task schedules.

Terminology

Diffusion Model — 1× in this paper · explained in 10 episodes
Diffusion Models — 1× in this paper · explained in 9 episodes

Summary

The gist The introduction focuses on online reinforcement learning for diffusion models, specifically addressing credit-assignment problems in reverse-process and forward-process methods, and introduces DiffusionART to achieve efficient and stable training without per-task tuning.

How it works

DiffusionART is introduced as a method that aggregates each prompt’s rewards into a shared reward-improving direction for reverse-process states, which removes the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning The algorithm combines supervised velocity regression from forward-process training with updates applied to states and actions during reverse-process rollouts

Key Mechanisms and Insights

DiffusionNFT instability: The paper diagnoses the instability of NFT-style training by showing that the objective implicitly contains a harmful forward–reverse consistency force that joint training cancels only partially This force arises because the positive and negative branches carry uncancelled consistency forces, where the positive branch pulls velocity toward vold with weight q, whereas the negative branch pushes it away with weight 1 − q

Restoration as a Stabilizer: The paper identifies restoration as the key stabilizer, showing that EMA stabilizes NFT through restoration rather than off-policy rollouts, but this stabilization requires task-specific tuning Proposition 2 demonstrates that the NFT loss can be decomposed into a linear term carrying both reward guidance and the consistency force, and a quadratic term restoring vθ toward vold

Aggregation-First Guidance: DiffusionART applies aggregation-first reward guidance to reverse-rollout states by aggregating each prompt’s samples and rewards into a shared reward-improving direction G(c) This is constructed by normalizing the covariance of rewards and endpoint means across all sampled endpoints for a given prompt, denoted as Cov(R, x0 c) in Equation 5

Two-Update Restoration: DiffusionART uses a second MSE update with a trust-region mask mechanism on each rollout to add a restoration force proportional to each state’s drift, replacing EMA with no schedule to tune This second update is defined as L(1) = w MSE(vθ(st), vART tgt (st)) = w D−1∥G(c) − d(st)∥ squared 2 where d(st) = vθ(st) − vold is the current displacement

Trust-Region Masking: A trust-region mechanism is used to constrain excessive policy changes by defining a KL proxy m(st) and masking updates where the outward drift is positive, as defined by w(st) = t[1 − 1if m(st) ≥ δ] 1if⟨d(st), G(c) − d(st)⟩ > 0

Performance and Scalability

DiffusionART demonstrates superior efficiency and stability across various generation tasks compared to baselines like Flow-GRPO-Fast and DiffusionNFT On HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02× and 11.72× fewer GPU hours, respectively For image-to-video generation, it reaches MQ 6 with 4.8× fewer GPU hours than Flow-GRPO-Fast DiffusionART trains efficiently and stably without per-task tuning, matching the strongest baseline’s efficiency on SD3.5-M without EMA or per-task tuning

Conclusion

DiffusionART systematically cancels the unstable consistency force by aggregating endpoint reward guidance before assigning it to reverse-rollout states, and it uses a second update with a trust-region mechanism to provide drift-adaptive restoration for stabilization Experiments on image and video generation show efficient, stable reward improvement, including with pairwise feedback and small rollout groups The work provides insights into forward-process training, offering a clearer understanding of diffusion reinforcement learning and a path toward scalable diffusion post-training for visual generation

Appendix Details

** Lemma 1 proves the conditional covariance identity for ideal NFT, showing that the positive and negative branches point along the same reward-tilt direction g Lemma 3 motivates a shared reward-improving direction computed from the K endpoints already sampled per prompt, which avoids state-conditional posterior sampling and excludes the NFT self-consistency term by construction The pairwise covariance identity shows that the global covariance equals the average of the M pair covariances, establishing a specialization for pairwise feedback The trust-region diagnostics show that restricting second updates helps keep later first updates well behaved, rather than merely suppressing gradients at the masked update The paper also details extensions to pairwise reward models and ablation studies on guidance and restoration components The experiments show that DiffusionART is insensitive to the group size K, and a smaller K substantially reduces GPU hours The paper also compares DiffusionART with ReFL under matched VQ:MQ = 1:1 weights, noting that DiffusionART improves both rewards with fewer optimizer updates while ReFL uses fewer samples per update The paper also shows that the second update in ART: 2 updates introduces restoration and provides long-term stability, in contrast to NFT: 2 updates which briefly collapses to near zero before recovering The paper concludes by showing that DiffusionART reaches training VQ of 6 in 20 steps, compared with 50 for DiffusionNFT (EMA) and 427 for Flow-GRPO-Fast The paper also shows that the mean reward over the final 50 training steps is 13.018, compared to 10.105 and 8.510 for the two baselines, representing improvements of 28.8% and 53.0% The paper also shows that DiffusionART improves evaluation VQ and MQ by 31.4% and 57.4%, respectively, over the strongest baseline The paper also shows that DiffusionART reaches training VQ of 6 in 20 steps, compared with 50 for DiffusionNFT (EMA) and 427 for Flow-GRPO-Fast: 2.5× and 21.4× fewer training steps The paper also shows that DiffusionART reaches MQ of 3 in 19 steps and 42.9 GPU hours, compared to 33 steps and 165.6 GPU hours for NFT, and 117 steps and 149.4 GPU hours for Fast The paper also shows that DiffusionART improves both VQ and MQ fastest while maintaining TA comparable to the strongest baseline The paper also shows that DiffusionART reaches MQ of 6 in 45.8 GPU hours versus 220.3 for Flow-GRPO-Fast, a 4.8× reduction The paper also shows that DiffusionART improves both VQ and MQ efficiently, whereas the baselines improve slowly or plateau The paper also shows that DiffusionART reaches MQ of 3 after 821 steps and 1805.5 GPU hours The paper also shows that DiffusionART improves both VQ and MQ efficiently, whereas the baselines improve slowly or plateau The paper also shows that DiffusionART reaches MQ of 6 in 45.8 GPU hours versus 220.3 for Flow-GRPO-Fast, a 4.

Improvements for AI systems

When fine-tuning diffusion models for text-to-image generation, DiffusionART can reach late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02× and 11.72× fewer GPU hours, respectively while achieving 28.8% and 53.0% higher mean training rewards.
For text-to-video generation, DiffusionART can surpass all baselines in both VQ and MQ while keeping TA comparable with the strongest baseline, reaching training VQ of 6 and MQ of 3 after 29 steps, demonstrating efficiency gains over methods like Flow-GRPO-Fast which require significantly more GPU hours.
The system can be trained efficiently without per-task tuning by using Aggregation-first guidance to aggregate each prompt’s rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction.
Training stability is enhanced through a second update with a trust-region mask mechanism that provides drift-adaptive restoration without EMA tuning, effectively replacing the need for task-specific EMA schedules.
DiffusionART can support online pairwise feedback through DiffusionART-Pairwise, where it computes guidance using pairwise rewards yi = 1 to winners, yi = 0 to losers, and yi = 1/2 for ties, allowing for stable optimization even with objective uncertainty.