SoftCache: Adaptive Step Size Caching for Diffusion Transformers

arXiv:2610.22369 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SoftCache: Adaptive Step Size Caching for Diffusion Transformers".

Jane: The gist Diffusion and flow models generate a sample over iterative steps, each a full pass through a large network,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper called "SoftCache: Adaptive Step Size Caching for Diffusion Transformers." The main idea here is that diffusion and flow models generate a sample over iterative steps, and each step is a full pass through a huge network, which takes up most of the generation cost.

Jane: That sounds like it's slow. So what are these authors trying to do to speed that up? What's the big claim they're making about this SoftCache thing?

Tom: They propose training-free caching that makes it adaptive instead of hard. Instead of a binary switch deciding whether to refresh or reuse activations, they use something called headroom below a staleness threshold.

Lu: So what does that mean for the math? How do they quantify this headroom and how does it translate into making the step size change?

Jane: That's the next thing I want to get at. If it's not just a switch, what is this headroom, and how does it actually influence where the sampler goes next?

Tom: It says that the signal below the threshold tracks how much error is induced by reuse. This distance from the threshold, which they call headroom Hi equals one minus Ai over tau, measures how safely we can trust the cache.

Meng: From a deployment standpoint, if it's soft caching instead of hard caching, what does that mean for production? Does it introduce instability when you're just trying to ship a model?

Tom: It means that where the headroom is large, the cache is still reliable so the sampler takes a long step that skips several scheduled nodes and can land between them. When the headroom gets small, refresh is near, so the step shrinks back toward the scheduled one.

Jane: So instead of just deciding to reuse or refresh at every single node on a fixed schedule, this method uses that distance from the threshold to decide how long each cached step should be. That sounds like it could make the sampling much more efficient than a simple on-off switch.

Lu: And they handle those jumps when they happen by extending the residual along a straight line through the two most recent fresh residuals to keep sampling off that fixed grid. They also re-space the tail so every node passed is dropped from the trajectory using a coordinate map that warps the landing node.

Tom: So, on paper, they show this approach has no per-backbone settings at all. That's important because it means you don't have to tune anything specific for images or video models when you implement SoftCache.

Jane: The results they show are pretty impressive too. They tested it on FLUX for images, HunyuanVideo and Wan2 point 1 for video, and they say that at speedups from three point five times up to six point nine times, SoftCache is ahead of every baseline on all four fidelity metrics.

Meng: Being ahead across that whole range of speedups is a lot. But what about the cost? Does this adaptive stepping actually make the per-step cost lower or higher in practice when you're running this on real traffic?

Tom: The paper focuses on the performance gains, showing speedups from three point five times to six point nine times. It doesn't explicitly detail the runtime cost of that adaptive calculation itself, but they show it beats existing methods by a factor of up to six point nine times.

Jane: So for someone just listening to this, what does this mean for how we use these massive generative models? What is the real-world implication here beyond just being faster on a benchmark?

Lu: This paper is about turning hard caching into soft caching by making the step size adaptive based on the reuse budget remaining. This adaptivity decides where the sampler goes next as well as when to refresh.

Tom: The authors point out that removing the jump mechanism costs a lot at seven times, which shows how critical that adaptive jumping part is for maintaining performance.

Jane: So, thinking about what they've done with "SoftCache: Adaptive Step Size Caching for Diffusion Transformers," what do you think is the main thing we should remember about this research?

Lu: The core contribution is identifying that the refresh signal of a dynamic cache carries more than just a binary decision. Its headroom below the threshold tracks the reuse error, so it measures how far the cached computation can still be trusted.

Tom: And they propose SoftCache which reads this headroom as a step size instead of just making a binary switch. That's the key mechanism.

Jane: It really shifts the focus from just deciding *if* you reuse to deciding *how much* you reuse based on how much error is expected. That sounds like it solves a problem where fixed schedules are too rigid for complex generative processes.

Meng: For deployment, having something that adapts its movement based on the current state of computation seems more robust than a fixed schedule that might miss an optimal sampling point.

Tom: Exactly. And they showed this works without any per-backbone settings, which is a big win for engineers trying to keep their pipelines simple.

Jane: So, in terms of what this paper means for the future of these models, it suggests that we can get better performance from caching by making the caching strategy itself smarter and more nuanced than just a simple on or off switch.

Conclusion: Tom: So, we're looking at this paper called "SoftCache: Adaptive Step Size Caching for Diffusion Transformers." The authors are tackling the problem of slow diffusion sampling by changing caching from a simple on or off switch to something more nuanced.

Jane: What does that actually mean for a user running these models? Is it just about speed, or does it change the quality of what you get?

Lu: It fundamentally changes the sampling trajectory. Instead of sticking to a fixed grid, the step size adapts based on how much trust we have in our cached data.

Meng: I’m curious about what this means for deployment. Does it add complexity to setting up a pipeline when you're trying to ship something reliable?

Tom: It actually makes the setup simpler because they say there are no per-backbone settings needed for image or video models, which is a big deal.

Jane: So, if you listen to this show and only get one thing about SoftCache, what should it be?

Lu: It’s that the method turns hard caching into soft caching by making the step size adaptive based on the reuse budget remaining.

Tom: Exactly. It's not just deciding whether to refresh or reuse; it's deciding how far each cached step goes. That adaptivity is what they call the key mechanism here.

Jane: It sounds like this moves beyond just optimizing a single model and starts rethinking the whole sampling strategy for these giant generative systems.

Meng: I’m still wondering about those performance numbers they show against the baselines, especially when you're running on real traffic where things aren't perfectly controlled.

Lu: They tested it across FLUX, HunyuanVideo, and Wan2 point one and they show speedups from three point five times up to six point nine times across all four fidelity metrics <ref:2610.22369#pg1>.

Tom: That range of speedup is quite impressive if that holds up when you put it in a production environment with unpredictable inputs.

Jane: It suggests that for future work, the focus isn't just on making one specific model faster, but on developing smarter, more nuanced sampling strategies for all these massive generative systems.

Shimon Malnick, Nadav Orenstein, Ohad Fried, Shai Avidan

Tel Aviv University · Reichman University

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: Project page: https://malnick.net/softcache

Code: https://github.com/black-forest-labs/flux

Project page: https://malnick.net/softcache

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist Diffusion and flow models generate a sample over iterative steps, each a full pass through a large network, and that pass dominates generation cost SoftCache proposes soft caching by using

Key concepts

Hard Caching: A simple caching method where the system makes a binary decision: either reuse cached results or perform a fresh network evaluation. Every step, regardless of the decision, advances by a fixed step size defined by the scheduler. This approach is less flexible than what SoftCache achieves.
Headroom (Hi): The headroom measures how much trust remains in a cached computation before it becomes unreliable. It is calculated as 1 minus the ratio of current evaluation time to a staleness threshold ($ au$). A larger headroom indicates that the cached step can be trusted for a longer duration, allowing for larger adaptive jumps.
Adaptive Step Size: SoftCache uses the headroom ($H_i$) to determine how long each cached step should be. Instead of fixed intervals, the step size ($ar{ riangle}_i$) scales with the headroom. When headroom is high, steps become large enough to jump off the fixed scheduler grid, potentially landing between scheduled nodes.
Trajectory Discontinuities Handling: When a sampler jumps to a new location due to an adaptive step, SoftCache corrects the trajectory by linearly extrapolating the residual along a straight line connecting the two most recent fresh residuals. It also re-spaces the tail of the trajectory to ensure every node passed during a jump is properly accounted for.

Terminology

Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist Diffusion and flow models generate a sample over iterative steps, each a full pass through a large network, and that pass dominates generation cost SoftCache proposes soft caching by using the headroom below the threshold to set an adaptive step size rather than a binary switch between refresh and reuse This method is ahead of every baseline on all four fidelity metrics at comparable speedups on FLUX, HunyuanVideo, and Wan2.1.

How it works

The core idea of SoftCache is to turn hard caching into soft caching by using the headroom below a staleness threshold to set the step size. Dynamic caches spend their signal on a single binary decision, but SoftCache reads how far below the threshold it sits, which measures how far the cached computation can still be trusted. This headroom is defined as Hi = 1 − Ai/τ, which lies in [0, 1] at a cached node.

The adaptive step size mechanism

SoftCache uses this headroom to set the length of the next step rather than just deciding whether to refresh or reuse. The formula for the new step size is defined as ∆¯i = min(1 − ti, ∆i + (J − 1)Hi∆i) when Ai < τ, which means cached steps jump off the scheduler’s grid in proportion to the headroom. When the headroom is large, the sampler takes a long step that skips several scheduled nodes and can land between them.

Handling trajectory discontinuities

When a jump occurs, SoftCache extends the residual along a straight line through the two most recent fresh residuals to continue sampling off the fixed grid. The new residual is defined as rbi = ra + β λi (ra − ra−1), where λi is built from time gaps. Furthermore, the tail is re-spaced so that every node passed is dropped from the trajectory by using a coordinate map that warps the landing node and spaces mi nodes evenly between it and t = 1.

Key contributions and performance

The main contributions of SoftCache are identified as follows:

We identify that the refresh signal of a dynamic cache carries more than a binary decision: its headroom below the threshold tracks the reuse error, so it measures how far the cached computation can still be trusted.
"We propose SoftCache, which reads the headroom as a step size: cached steps jump off the scheduler’s grid in proportion to it, the tail is re-spaced so that every node passed is dropped from the trajectory, and the cached residual is extrapolated linearly to the landing".
On image and video backbones, SoftCache adds no perbackbone settings and is ahead of every cache we compare on all four fidelity metrics in every speed band from 3.5× to 6.9×.

Experimental results

SoftCache has been evaluated on FLUX, HunyuanVideo, and Wan2.1 across various speed bands ranging from 3.5× to 6.9× The method is shown to be ahead of every baseline on all four fidelity metrics in every speed band. For instance, on FLUX at the fastest band, SoftCache leads SeaCache by 1.12 dB PSNR and 0.082 LPIPS. On HunyuanVideo, SoftCache leads both SeaCache and DPCache on all four metrics in all three bands by 0.54 to 2.78 dB PSNR over the better one.

Ablations and analysis

A series of ablations were performed to isolate the effects of different components, such as removing the jump or headroom scaling. The results show that Removing the jump costs the most at 7×, while Removing headroom scaling or residual motion behaves differently. Qualitative comparisons on FLUX show that SoftCache keeps every object the prompt names, with its shape intact, in all three rows.

The paper concludes that SoftCache is a training-free method that turns hard caching into soft caching by making the step size adaptive based on the reuse budget remaining. This adaptivity decides where the sampler goes next as well as when to refresh. The final result demonstrates that SoftCache stays closest to the uncached sample on all four fidelity metrics, with no per-model calibration.

Improvements for AI systems

Improved inference latency by implementing SoftCache, which performs soft caching: instead of a binary switch, the headroom sets how far each cached step goes. This allows samplers to jump past several scheduled nodes when the cache is reliable, effectively taking longer steps and landing between scheduled nodes, which translates to speedups from 3.5× to 6.9× on FLUX and video models.
Enhanced fidelity across all metrics by utilizing SoftCache's adaptive step size control, where the headroom sets how far each cached step goes, ensuring that cached steps jump off the scheduler’s grid in proportion to it and the tail is re-spaced so that every node passed is dropped from the trajectory. This results in SoftCache reproducing the uncached model most faithfully on every metric at every operating point tested.
Increased robustness against sampling errors by incorporating explicit jump guards into SoftCache, which are defined as thresholds: the opening stretch, ti 0.18. These guards ensure that critical nodes take the scheduled step instead of an adapted one when the cached residual might be unreliable.
Improved generalization across different sampling schedules by allowing SoftCache to dynamically adjust its step size based on how far below the threshold it sits, effectively deciding where the sampler goes next rather than just when to refresh, leading to steps that shrink back toward the original step as a refresh nears.
Increased efficiency by leveraging residual extrapolation during off-grid landings, where SoftCache extends cached residuals using a straight line through the two most recent fresh residuals, thereby avoiding recomputation while maintaining trajectory accuracy.

Abstract

Diffusion and flow models generate a sample over iterative steps, each a full pass through a large network, and that pass dominates generation cost. Training-free caching accelerates them by reusing activations from recent evaluations instead of recomputing them. Dynamic caching methods decide when to refresh by accumulating a cheap probe of the network input into a staleness signal and spending a fresh evaluation once it crosses a threshold. They perform hard caching: the signal is a binary switch between refresh and reuse, and every step, fresh or cached, advances by the prescribed step size. We observe that the signal carries much more than this switch. How far it sits below the threshold, its headroom, predicts the error induced by reuse. We propose SoftCache, which performs soft caching: instead of a binary switch, the headroom sets how far each cached step goes. Where the headroom is large, the sampler jumps past several scheduled nodes, and as a refresh nears, its steps shrink back toward the original step. These adaptive jumps may land between the scheduled nodes, taking the sampler off its fixed grid. SoftCache requires no training and adds no per-backbone settings: on FLUX for images and on HunyuanVideo and Wan2.1 for video, at speedups from 3.5 times to 6.9 times, it is ahead of every baseline on all four fidelity metrics.