Consistency Deep Equilibrium Models

arXiv:2602.03024 · cs.LG, cs.AI · Submitted 2026-02-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Consistency Deep Equilibrium Models".

Jane: The gist Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage;

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Alright everyone, let's talk about this paper, "Consistency Deep Equilibrium Models." The main idea is that deep equilibrium models can be powerful because they model infinite depth with constant memory usage, but they get slow during inference because of those iterative fixed-point solvers.

Jane: So, what does this paper claim to fix? It introduces something called the Consistency Deep Equilibrium Model or C-DEQ, which uses consistency distillation to speed up that inference process.

Lu: Basically, they reframe the iterative DEQ process as evolution along a fixed ordinary differential equation trajectory toward the equilibrium. They then train these C-DEQs to map intermediate states directly to that final fixed point, letting you do few-step inference while keeping the performance of the original teacher model.

Meng: So it's about taking that slow iterative process and turning it into something more direct by using this ODE trajectory idea. But what's the actual mechanism behind how it does this?

Tom: Well, they define this as path-independent Deep Equilibrium Models as trajectory-fixed Fixed Point ODEs or FP-ODE defined as dzt/dt equals G theta of z t and t. That formulation picks a deterministic trajectory for consistency distillation and keeps the same equilibrium that the original DEQ has, which is key for making it work.

Jane: That ODE perspective seems important because consistency distillation is usually designed for continuous-time dynamical systems, so this framing makes it a principled way to handle these fixed points.

Lu: Exactly. By reframing the DEQ inference as solving a fixed FP-ODE whose terminal state approaches the equilibrium, they get a principled and deterministic path that's really good for consistency distillation because it's defined by continuous-time dynamics.

Tom: Then they generate this teacher trajectory using Anderson Acceleration or AA, which is used to efficiently use historical iterates to produce stable intermediate states. They construct this trajectory T from zero through K, where z K serves as an approximation of the equilibrium z star.

Jane: And to help the student learn along that path, they incorporate structural priors from the Anderson Acceleration solver into the student model so it can refine those intermediate estimates instead of just rediscovering the solver's trajectory itself.

Meng: So this means we don't need a massive number of steps to get close to that equilibrium anymore because we have a guided path for the model to follow. But how do they actually build this C-DEQ model, what is the structure?

Tom: The C-DEQs are parameterized by g phi of z less than or equal to t and t and x, which in practice means they can take either just the current state z t or a short window of past iterates.

Paper summary: Jane: And they define g phi as cskip of t z t plus cout of t P phi of z less than or equal to t and x, where cskip is (T minus epsilon over gamma) and cout is one minus T minus epsilon over gamma, with gamma being greater than or equal to one.

Lu: That structure is interesting because it means the model only makes minor corrections near convergence, but for the early solver states at t equal to epsilon, the prediction is dominated by that neural network P phi, which lets it learn the global mappings needed to align initial iterates with the fixed point.

Tom: To train it, they use a distillation loss with two parts: a global consistency loss and a local consistency loss. The global consistency loss anchors predictions to the equilibrium using a distance metric like MSE against z K, and then they have this local consistency loss that makes predictions invariant across adjacent points along the solver's trajectory.

Jane: So you’ve got both anchoring to the final goal and making sure the model flows smoothly from one intermediate state to the next along that specific trajectory. That sounds like a solid way to ensure stability while aiming for accuracy.

Meng: If I'm looking at this from a production standpoint, having both losses means we need to make sure our training setup can handle that dual constraint without it blowing up or becoming too computationally expensive during the actual training run.

Tom: And they found that combining these objectives is necessary for getting accurate and stable few-step inference, and they used a balanced mixture with lambda one equal to zero point six as a good example of how to balance global and local objectives.

Jane: So what does this mean practically for someone who isn't deep in the math? It means we can get two to twenty times more accuracy improvements over implicit DEQs using the same few-step inference budget, which is pretty significant <ref:2602.03024#pg1,the same few-step inference budget>.

Lu: That means they show that C-DEQs achieve consistent two to twenty times accuracy improvements over implicit DEQs under the same few-step inference budget, while matching or surpassing explicit finite depth networks and narrowing the latency gap between implicit and explicit models <ref:2602.03024#pg1,accuracy improvements over implicit DEQs under the same few-step inference budget>.

Meng: How does that translate to what I deal with daily? If I'm running a model on real traffic, what's the cost of this C-DEQ versus just running a standard DEQ solver?

Tom: They show that compared to DEQ and HyperDEQ at one step, the inference time of C-DEQ significantly outperforms other DEQs up to fourteen point six times while performance is achieving up to five point three times improvement.

Paper summary: Jane: That comparison against the explicit state-of-the-art Transformer with eight steps shows an absolute difference of only two point one in perplexity, which is a small gap compared to the gains they achieve with C-DEQ.

Lu: The ablation studies on Anderson Acceleration show that by explicitly using solver history, C-DEQ approaches the converged baseline in only five to six steps, which is much faster than the non-AA variant that needs nearly two times the steps.

Meng: So it seems like incorporating that solver history isn't just a nice theoretical thing; it actually cuts down the required steps by almost half if you use AA instead of just standard methods.

Tom: And they found that the optimal configuration for those loss coefficients is lambda one equal to zero point eight and lambda two equal to zero point zero five with a perplexity of twenty-five point zero, which validates this composite loss design really well.

Jane: So the overall implication here is that C-DEQ effectively accelerates the convergence process into only a few function evaluations, yielding accuracy gains as inference steps increase.

Lu: The paper states that they propose C-DEQ, a framework that accelerates DEQ inference by distilling along a fixed FP-ODE trajectory. By leveraging AA structural prior and balancing global and local consistency, C-DEQs map intermediate states directly to the equilibrium, enabling few-step inference while retaining the benefits of iterative solving.

Meng: The authors admit that their limitation is that they are distilling along a pretrained DEQ teacher, meaning you still have to train the original teacher model first before you can use this acceleration framework.

Tom: That's a fair point, and they say the FP-ODE view mainly serves as a trajectory selection perspective; stronger theoretical guarantees and adaptive solver-induced trajectories are left for future work.

Jane: So to wrap up, the Consistency Deep Equilibrium Models paper introduces C-DEQ, which uses consistency distillation along a fixed FP-ODE trajectory to map intermediate states directly to the equilibrium. This allows for few-step inference while keeping the performance of iterative solving methods.

Tom: The main point is that they show consistent two to twenty times accuracy improvements over implicit DEQs under the same few-step inference budget, and this narrows the efficiency gap between implicit DEQs and explicit networks <ref:2602.03024#pg1,accuracy improvements over implicit DEQs under the same few-step inference budget>.

Lu: This work confirms that by using a guided path defined by solver history and balancing global and local consistency, you can get these accuracy improvements while compressing the iterative process for language modeling.

Meng: For someone shipping models, it means we have a framework that can compress the iterative solving process into just a few function evaluations without losing too much performance.

Jane: It really shows how important it is to look at these equilibrium models through this lens, showing them to be much more efficient tools for practical applications than they used to be.

Conclusion: Tom: So we’ve been looking at Consistency Deep Equilibrium Models, or C-DEQ. It’s essentially this new way to make those deep equilibrium models faster during inference using consistency distillation along a fixed ODE trajectory.

Jane: Right, so the authors are taking these inherently slow iterative solvers and re-framing the process as evolution along a deterministic path toward the fixed point. That sounds like they're trying to guide the model instead of letting it wander aimlessly.

Lu: Exactly, they define this as a fixed point ODE where dzt over dt equals G theta of z t and t, which gives them that deterministic trajectory to work with for their distillation process. It’s about picking a specific path rather than just hoping the solver finds the right answer by itself.

Meng: But what does this actually mean for someone trying to deploy a model? Is it just theoretical acceleration, or is it something that actually makes running these massive models feasible?

Tom: Well, the experiments show they get consistent two to twenty times accuracy improvements over implicit DEQs while using the same few-step inference budget. That’s a big jump when you’re looking at practical deployment constraints.

Jane: So, for the listener driving right now, it means that if you use this C-DEQ method, you can get much better results without having to run hundreds of function evaluations on your server. It narrows the gap between these complex implicit models and simpler explicit networks.

Lu: And their ablation studies with Anderson Acceleration show that by explicitly using solver history, they cut down the required steps from nearly two times to just five or six steps for convergence. That structural prior really helps stabilize things.

Meng: I’m curious about the cost of this distillation step itself. If we have to train a separate teacher model first, how much extra training compute are we talking about before we can even start using the C-DEQ acceleration?

Tom: The paper notes that they need to train the original DEQ teacher, but their main point is that this FP-ODE view is primarily a way to select the right trajectory. Stronger theoretical guarantees on adaptive trajectories are still coming down the road.

Jane: So while it’s a powerful acceleration tool for inference, it still depends on having that initial teacher model trained first. That’s an important caveat to keep in mind for anyone considering this technique.

Lu: And they found that balancing the global consistency loss and the local consistency loss with a lambda one of zero point eight gives them the best performance-efficiency trade-off overall. It’s a solid configuration for learning how to map those intermediate states correctly.

Meng: So what we need to keep in mind is that it’s not magic; it relies on training a pre-existing model, but the resulting inference speed gains are substantial enough to investigate further if the deployment pipeline allows for that extra training phase.

School of Electronic Information and Communications, Huazhong University of Science and Technology · School of Science, Wuhan University of Technology

cs.LG, cs.AI

Submitted: 2026-02-03

Updated: 2026-05-31

Journal ref: Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:74652-74669, 2026

Code: https://github.com/landrarwolf/CDEQ

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: The gist Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage; this work introduces

Key concepts

Deep Equilibrium Models (DEQs): These are deep learning models designed to represent infinite-depth networks using constant memory. They achieve this by iteratively solving fixed-point equations, which is powerful but computationally slow during inference.
Consistency Deep Equilibrium Model (C-DEQ): A novel framework that accelerates DEQ inference by using consistency distillation. It reframes the iterative process as following a fixed ODE trajectory and trains the model to consistently map any intermediate state directly to the final equilibrium point, allowing fast, few-step inference.
Fixed Point ODE (FP-ODE): The iterative DEQ process is cast as evolution along a deterministic Ordinary Differential Equation (ODE) trajectory. This formulation defines a specific path for the solver dynamics, which is crucial because consistency distillation works best when applied to continuous-time dynamical systems like this ODE.
Anderson Acceleration (AA): A technique used to efficiently generate stable and informative intermediate states along the DEQ solver's trajectory. AA generates the next iterate by taking a linear combination of past iterates, ensuring the teacher trajectory is smooth and well-defined.

Terminology

Consistency — 3× in this paper · explained in 3 episodes
Accuracy — 1× in this paper · explained in 1 episode
Consistency Distillation — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Fixed Point — 1× in this paper · explained in 2 episodes

Summary

The gist Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage; this work introduces the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference.

How it works

The C-DEQ framework reframes the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium; along this trajectory, C-DEQs are trained to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. This is achieved by casting path-independent DEQs as trajectory-fixed Fixed Point ODEs (FP-ODE) defined as dzt/dt = Gθ(zt, t), where Gθ is a conceptual function representing the solverinduced dynamics; this formulation selects a deterministic trajectory for consistency distillation and preserves the same equilibrium as that of the DEQ in Eq. (1); this ODE perspective for DEQs is crucial for enabling consistency distillation which is inherently defined on continuous-time dynamical systems; by reframing DEQ inference as the numerical solution of a solver-induced FP-ODE whose terminal state approaches the equilibrium, we obtain a principled and deterministic path that is particularly well suited for consistency distillation.

Teacher Trajectory and Consistency Distillation

To generate the teacher trajectory, Anderson Acceleration (AA) is employed to efficiently exploit historical iterates to produce stable and informative intermediate states; AA iteration generates the next iterate zk+1 as a linear combination of the past m iterates and their function evaluations; this teacher trajectory T = 0, which is constructed sequentially from k = 0 to K; where S AA is defined by S AA(zk−mk+i) mk i=0 fθ, x (3); the complete procedure for AA is detailed in Algorithm 3 of the Appendix C.1; where zK serves as an approximation of the equilibrium z⋆. To facilitate learning along this trajectory, we apply standard consistency parameterization and, importantly, incorporate the structural prior of the Anderson Acceleration solver (Walker & Ni, 2011) into the student model; this allows the student to refine intermediate estimates rather than rediscover the solver trajectory, accelerating convergence and preserving solver-like stability.

C-DEQ Parameterization and Inference Procedure

The C-DEQs are denoted by gϕ(z≤t, t, x) parameterized by learnable parameters ϕ; where z≤t denotes the history of solver states up to t; in practice, this can be instantiated as the current state zt alone or a short window of past iterates; the primary objective is to construct a model gϕ that consistently maps intermediate states along the same trajectory to the equilibrium. To ensure the model is anchored at the equilibrium end, we introduce a free-form parameterized network Pϕ(z≤t, t, x) whose output has the same dimensionality as zt; and define gϕ(z≤t, t, x) = cskip(t)zt+cout(t)Pϕ(z≤t, t, x), t ∈ [ϵ, T), with cskip(t) = (T − ϵ)/γ and cout(t) = 1 − (T − ϵ)/γ; with γ ≥ 1. As such, the model naturally makes only minor corrections near convergence; conversely, for early solver states (t ≈ ϵ), the prediction is dominated by the neural network Pϕ, allowing the model to learn the global mappings required to align initial iterates with the fixed point.

Consistency Deep Equilibrium Models Training Objective

The training objective of C-DEQ consists of two complementary terms designed to ensure that the student model hϕ is both stable and accurate; specifically, given an input x and a sampled time index k ∼ U U(1, K) from the trajectory T, we define the distillation loss as a combination of local and global constraints: Ldistill = λ1Lglobal + (1 − λ1)Llocal + λ2Ltask (11); the global consistency loss is Lglobal(ϕ; θ):= E t∼p(t) d gϕ(ztk, tk, x), zK, where d(·, ·) is a distance metric (e.g., MSE) and zK is the equilibrium produced by the teacher solver; while local consistency loss enforces model predictions to be invariant across adjacent points along the solver’s trajectory, defined as Llocal(ϕ, ϕ −; θ):= E t∼p(t) d gϕ(ztk, tk, x), gϕ− (ztk−1, tk−1, x); by aligning the estimates from a state ztk with its immediate predecessor ztk−1, we encourage the consistency function to learn a smooth flow; combining both objectives is therefore necessary to achieve accurate and stable few-step inference.

Experimental Results and Efficiency Gains

Extensive experiments across various domain tasks demonstrate that C-DEQs achieve consistent 2-20× accuracy improvements over implicit DEQs under the same few-step inference budget; in Table 1, C-DEQ achieves substantial perplexity improvements without requiring a large number of function evaluations; specifically with one step, C-DEQ attains a perplexity that is within an absolute gap of only 2.7 to the explicit TCN model. Compared to the explicit state-of-the-art Transformer with 8 steps, there is only an absolute difference of 2.1 in perplexity; furthermore, compared to DEQ and HyperDEQ at one step, the inference time of C-DEQ significantly outperforms other DEQs up to 14.6×, while performance achieving up to 5.3× improvement; this result suggests that our multi-step sampling strategy effectively rectifies the solver trajectory and substantially compresses the iterative process for language modeling.

Ablation and Robustness

Ablation studies on AA show that by explicitly leveraging solver history, C-DEQ (red) approaches the converged baseline (dotted) in only 5–6 steps, significantly faster than the non-AA variant (green) which requires nearly 2× the steps; this confirms the necessity of the AA parameterization. Ablation on Solver Choice reveals that AA provides the best performance-efficiency trade-off across tasks; similarly, ablation on loss coefficients shows that a balanced mixture (e.g., λ1 = 0.6) significantly outperforms relying solely on global (λ1 = 1) or local (λ1 = 0) objectives; overall, the optimal configuration is achieved at λ1 = 0.8 and λ2 = 0.05 with a PPL of 25.0, validating the effectiveness of our composite loss design. The results suggest that C-DEQ effectively accelerates the convergence process into only a few function evaluations, yielding accuracy gains as inference steps increase; this confirms that our C-DEQ is a general and outstanding acceleration framework that scales to complex graph data and mitigates the inference bottleneck of iterative solvers on large-scale graphs.

Conclusion

We proposed C-DEQ, a framework that accelerates DEQ inference by distilling along a fixed FP-ODE trajectory; by leveraging the AA structural prior and balancing global and local consistency, C-DEQs map intermediate states directly to the equilibrium, enabling few-step inference while retaining the benefits of iterative solving. Extensive experiments across vision, language, and graph tasks demonstrate that C-DEQs achieves consistent 2-20× accuracy improvements over implicit DEQs under the same few-step inference budget; effectively narrowing the efficiency gap between implicit DEQs and explicit networks. The limitations include that C-DEQ accelerates inference by distilling a pretrained DEQ teacher, but still requires training the original teacher model; however, our FP-ODE view mainly serves as a trajectory-selection perspective; stronger theoretical guarantees and adaptive solver-induced trajectories are left for future work.

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning; there are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Acknowledgements

Z. Ling would also like to acknowledge the National Natural Science Foundation of China (NSFC-62406119), the National Key Research and Development Program of China (No. 2025YFA1018600), the Natural Science Foundation of Hubei Province (2024AFB074), and the Fundamental Research Support Program of HUST (2025BRSXB0004); R. C. Qiu is partially supported in part by the National Natural Science Foundation of China (via NSFC-12141107), the Key Research and Development Program of Wuhan (2024050702030100), and the Key Research and Development Program of Guangxi (GuiKe-AB21196034).

References

Anderson, D. G. Iterative procedures for nonlinear integral equations. Journal of the ACM (JACM), 12(4):547–560, 1965; Anil, C., Pokle, A., Liang, K., Treutlein, J., Wu, Y., Bai, S., Kolter, J. Z., and Grosse, R. B. Path independent equilibrium models can better exploit test-time computation. Advances in Neural Information Processing Systems, 35: 7796–7809, 2022; Bai, S., Koltun, V., and Kolter, J. Z. Multiscale deep equilibrium models.

Improvements for AI systems

Bold C-DEQ Inference Acceleration: The C-DEQ framework achieves state-of-the-art acceleration, bringing fixed-point inference into the “one-step” era, with up to 20× faster inference while maintaining the teacher’s performance. This allows for real-time natural language processing tasks where low latency is critical.
Bold Multi-Step Inference Flexibility: The system facilitates multistep evaluation to flexibly trade computation for performance gains, enabling users to trade computation for accuracy by sequentially applying the consistency mapping over multiple time steps.
Bold Enhanced Graph Learning Efficiency: For graph data, C-DEQ achieves significant speedups, such as being 'substantially faster than HyperDEQ (0.87s vs. 2.90s per batch) while being comparable to DEQ/MDEQ in runtime at NFE=8 on ImageNet classification and showing consistent 2-20× accuracy improvements over implicit DEQs under the same few-step inference budget.
Bold Robust Solver Utilization: By incorporating the "structural prior of the Anderson Acceleration solver (Walker & Ni, 2011) into the student model, C-DEQs allow for refine intermediate estimates rather than rediscover[ing] the solver trajectory, which preserves solver-like stability."
Bold Optimized Training Stability: The combined loss function, Ldistill = λ1Lglobal + (1 − λ1)Llocal + λ2Ltask, where a balanced mixture of global and local consistency losses is optimal at λ1 = 0.8 and λ2 = 0.05, ensures that the model is both stable and accurate, preventing issues like degenerate solutions while achieving accurate and stable few-step inference.
Bold Task-Aware Regularization: The incorporation of the task-level regularizer Ltask = E t∼p(t) CE(h(hϕ(zt, t)), y) ensures that intermediate states retain task-relevant information, leading to a consistent performance gain across all λ1 settings by preventing the model from solely focusing on solver dynamics.