The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks

arXiv:2610.21748 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "The Dormant Spiking Neuron".

Jane: The Dormant Spiking Neuron (DSN) model introduces a state-driven mechanism to address computational overhead in spiking neural networks by allowing neurons to enter a dormant state when their membrane potential…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper now called "The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks." Jane, what's the main idea here in plain English?

Jane: Well, Tom, the core idea is introducing a new way for spiking neural networks to save computation by letting neurons just stop working when they get too quiet. They call it the Dormant Spiking Neuron model and it uses a state-driven mechanism where if a neuron's membrane potential drops below some negative threshold, it enters a permanent dormant state, so you don't have to run all those calculations for that neuron anymore.

Tom: That sounds like they are targeting the continuous subthreshold dynamics that we always talked about being inefficient in SNNs. Jane, what are the specific numbers they found regarding this inefficiency?

Jane: They found that in a ResNet-eighteen model trained on ImageNet, up to forty-four percent of neurons end up in this deeply inhibited state where their membrane potential Vm is less than or equal to zero, even though they have a very small chance of firing soon <ref:2610.21748#pg1,up to 44% of neurons>. That's the computational burden they are trying to fix.

Tom: Forty-four percent is a lot of neurons just doing work that doesn't lead to any output. What does this paper actually propose as the solution for this problem?

Jane: The solution is a state-driven dynamic gate, which they call Dt n, and this gate updates based on the previous membrane potential. If that potential falls below a threshold Dth, the gate closes permanently and the neuron becomes dormant, meaning all subsequent stages of calculation are bypassed for that neuron entirely.

Tom: Bypassing all subsequent stages is key there. So they're not just pruning connections or neurons based on what they do during training; they're pruning based on their state while running inference. What does this dynamic gate do in terms of reducing the workload?

Jane: It allows for a multiplicative reduction in overall computational load because it operates independently from conventional input-driven sparsity, which is another way to prune networks. The authors show that by targeting this specific source of inefficiency, they can reduce the operations per neuron, or SOPsDSN,n to be lower than either input-driven or structural sparsity methods alone.

Title and authors: Tom: That's a big claim about multiplicative reduction. How much actual energy savings are we talking about in practice? Are these savings just theoretical for now?

Jane: They show that the DSN framework consistently reduces computational energy by forty percent to over seventy percent across both static and neuromorphic datasets. And on ImageNet, they found it actually outperforms other static pruning techniques at comparable reported cost reductions while simultaneously improving the baseline SNN accuracy by one point thirty-four percent.

Tom: Improving accuracy while cutting down computation is what we've been chasing for a long time. What about how this works during training? Since the dormancy gate involves a step function, that's not differentiable, so how do they actually train this thing?

Jane: They use backpropagation through time with surrogate gradients. This means they take the exact function in the forward pass but use a continuous approximation for its derivative when going backward during training to teach the network how to learn a robust dormancy policy.

Tom: Surrogate gradients are a necessary trick, I guess, to make things differentiable when you're dealing with those hard step functions. What does this mean for deployability? Can we just run inference and it works?

Jane: They offer two paths: inference-time optimization, which is like a plug-and-play tool you apply after training, and direct training through backpropagation through time using these surrogate gradients to learn the policy end-to-end.

Tom: So they can train it to learn the best way to become dormant based on what the data looks like. What kind of data are they looking at for this learning?

Jane: They found that when looking at static datasets, the dormancy threshold Dth stays around negative zero point five, which suggests a relatively stable low-energy operating point. The Jaccard similarity of their dormant masks shows high heterogeneity across different inputs, which means the model is leveraging instance-level redundancy rather than just redundancy across the whole dataset.

Tom: So it's not just pruning things that look the same everywhere; it's pruning based on what's happening in that specific instance. What about those models like transformers or SD-Transformers? Did they test those interactions?

Title and authors: Jane: Yes, they tested it, and applying the DSN to a transformer architecture consistently yielded substantial energy savings with only a limited impact on accuracy. It suggests this mechanism is quite versatile and doesn't just apply to standard CNNs.

Tom: Okay, so we have the results on efficiency gains and accuracy improvements. Before we wrap up this discussion, Lu, what’s your take on the actual structure of this DSN model? What would you change or build next?

Lu: I think the core idea is strong because it moves beyond just static pruning; it introduces dynamic sparsity based on state, which is a more intelligent way to handle temporal redundancy. If you could push this further, I'd look at how to make the dormancy threshold Dth itself adaptive based on the input context rather than just a fixed value.

Meng: From an engineering standpoint, forty percent to seventy percent savings is what matters for deployment cost and latency on real hardware. But I have to ask about scale; we know these models are trained on ImageNet, but how does this translate when we move to massive video streams or much larger models? Right now, the training setup seems quite specific.

Lalam: Based on the literature I've reviewed, the DSN framework is quite robust because it learns a policy directly. It’s sure that this mechanism works for reducing redundant computation in SNNs where subthreshold dynamics are an issue. I'm confident in its ability to learn a policy that preserves critical information flow during training.

Jane: So we've talked about the mechanics, the training, and those solid efficiency numbers. To wrap things up, what does this paper mean for how we design future spiking neural networks?

Tom: It means we have a way to handle those persistent subthreshold updates in SNNs without having to keep running all that computation continuously. It shows that state-driven sparsity is a viable orthogonal approach to input-driven sparsity.

Jane: Exactly, it provides a trainable mechanism for instance-level pruning, which is much more flexible than traditional methods because it learns what's truly redundant for a specific input scenario.

Tom: So we're looking at the paper "The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks" and they've shown how to reduce energy consumption by forty percent to over seventy percent, while maintaining competitive accuracy. That’s what it is.

The paper's summary: Tom: So, to recap, we're talking about how they introduced this state-driven mechanism for spiking neural networks to stop neurons from wasting energy when they get too quiet because their membrane potential drops too low.

Jane: Exactly. The main thing is that instead of keeping every neuron running continuously even when it's just drifting below a certain point, the DSN model lets them enter a dormant state permanently. It’s about dynamic pruning based on the actual neuron state, not just looking at what went into it from the outside.

Tom: I like that part about instance-level pruning. It means we aren't just cutting connections or neurons because they look similar across the whole dataset; we're cutting them because *this specific* neuron isn't doing anything important right now during a specific task.

Jane: That’s the core of it. They found that in deep models like ResNet-eighteen up to forty-four percent of neurons are just sitting there deeply inhibited, and this DSN mechanism targets that exact inefficiency by making them dormant.

Tom: And the numbers on efficiency are pretty compelling. They report energy reductions between forty percent and over seventy percent across both static and neuromorphic data sets. That’s a significant drop in computational cost without sacrificing much accuracy on ImageNet, actually improving it by one point thirty-four percent.

Jane: It really shows that you can achieve those massive hardware savings while keeping the model performing well, which is what people want when moving these models onto real neuromorphic chips. But they did mention a big hurdle for training because of that "dormancy gate" function.

Tom: Right, the non-differentiable step function. That’s where things get tricky for learning how to build this thing end-to-end, which is what they tackle with surrogate gradients during backpropagation through time. They use a smooth approximation of the step function to let the AI learn how to decide when a neuron should shut down.

Jane: It allows the network to learn this robust policy, meaning it learns *when* and *why* a neuron should stay dormant, rather than just having that decision hard-coded beforehand. It’s training the system to be smart about its own computation.

Tom: So the implication here is that we can build SNNs where they actively manage their own computational load based on real-time dynamics, making them much more resource efficient for deployment. But they did put a caveat about what this doesn't solve—it’s focused on instance-level redundancy, so maybe it doesn't handle things that are redundant across the entire dataset in the same way.

Jane: That’s fair. It’s a powerful tool for pruning based on transient behavior, but it might not capture every single type of redundancy that exists in a massive static dataset. Still, if you can get this to work reliably on complex video streams or anything that is nonstationary, this mechanism could be really useful for real-time AI applications.

Tom: It’s definitely something we need to watch closely since it provides a way to tackle those persistent subthreshold dynamics in SNNs without just running everything continuously. It’s a practical tool for efficiency.

The paper's improvements: Tom: So, moving on from how they train it, what are the actual improvements they suggest for using this DSN framework in practice?

Jane: Well, one major suggestion is to use inference time optimization or ITO after training, which means you can take that learned dormancy policy and apply it to a model right before you run it on real data. It's a plug-and-play tool that makes the savings immediately available without retraining anything.

Tom: That’s practical. It separates the learning phase from the deployment phase, which is always something we like when shipping things to production. But they also explore how this works with different types of AI architectures, right?

Jane: They tested it on transformer architectures and found that applying the DSN mechanism there still yields substantial energy savings even while keeping accuracy relatively stable. It suggests this isn't just a fix for standard convolutional neural networks either.

Tom: That’s interesting because if you can apply this state-driven sparsity to models like transformers, it means we can start seeing these efficiency gains in much larger, more complex AI systems that are currently too costly to run efficiently.

Jane: And they also looked at how the dormancy threshold changes depending on the data. They found that for static datasets, a threshold around negative zero point five works well for low-energy performance. It’s not a one-size-fits-all number; it depends on what you are trying to learn and what kind of data you have.

Tom: That dependency on the input context is key. It means the system is adapting its sparsity strategy dynamically, which addresses that limitation we talked about earlier where static pruning doesn't account for transient data patterns.

Jane: Exactly. The Jaccard similarity of their dormant masks showed high heterogeneity across different inputs, meaning it’s targeting specific instances rather than just general features in a dataset. This is what makes it more robust for real-world, nonstationary data streams.

Tom: So the implication for anyone building an SNN is that you can design a system where the neurons themselves become part of the pruning strategy through this state-driven mechanism, not just relying on external input filtering.

Jane: It gives us a way to make neuromorphic computing much more energy efficient by turning those continuous subthreshold dynamics into controlled dormancy states. But they did flag one limitation: they focused heavily on rate coding and multi-spike architectures, so it might not be as effective for time-to-first-spike coding where the information is encoded in the timing itself.

Tom: That’s a fair point about timing; if you're looking at pure temporal encoding, this mechanism might not capture all the necessary information flow. But for general pattern recognition on static images or sensor data, it seems like a solid way to go.

Conclusion: Tom: So, to wrap up on "The Dormant Spiking Neuron: A State-Driven Mechanism for Efficient Spiking Neural Networks," this paper basically shows how we can make spiking neural networks much more efficient by letting neurons enter a permanent dormant state when they get quiet.

Jane: That’s right. The big idea is using a dynamic gate to prune computations at the instance level, which means we don't have to run everything continuously anymore. It’s about smart, state-driven pruning instead of just static filtering based on input features.

Tom: Yeah, it gives us this multiplicative reduction in cost because it works alongside input-driven sparsity, which is a really clean way to save operations per neuron. The numbers showing forty percent to over seventy percent energy reduction are pretty solid when compared to other methods like static pruning.

Jane: And they didn't just cut costs; they actually improved the accuracy on ImageNet by one point thirty-four percent, which is something you don't see often when you’re trying to make a model smaller or faster.

Tom: It really changes how we think about deploying SNNs on hardware because it tackles that problem of continuous subthreshold updates without having to keep running all the calculations for neurons that aren't doing anything important.

Jane: It also gives us a way to train these models end-to-end using surrogate gradients, which is a clever trick to handle those non-differentiable step functions involved in the dormancy mechanism. That lets the AI learn its own policy for when to shut down.

Tom: So it’s trainable sparsity based on state, which is a really powerful concept for making models more resource efficient for real-world use cases. But they did mention that it might not work as well with purely time-to-first-spike coding because the information is encoded in when the spike happens, not just how often it happens.

Jane: That's a fair caveat. The DSN framework is definitely strong for tasks where you're looking at static input patterns, but we need to see if it holds up when the data itself is constantly changing and nonstationary.

Tom: Well, that’s what we’re thinking about next. I want to talk about how other papers are tackling those dynamic problems in a completely different way, like using mixture-of-experts structures for post-traumatic epilepsy diagnosis or some of those new memory construction methods for LLM agents.

Jane: Yeah, next up we look at things that are trying to manage complexity by splitting the model into functional experts instead of just pruning individual neurons.

Ruichen Ma, Yicai Chen, Pujun Zhou, Yue Zuo, Chongyang Li, Shuang Liu

University of Electronic Science and Technology of China (UESTC) · Shenzhen Institute for Advanced Study, UESTC

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Comments: NeurIPS 2026 Accepted

Code: https://github.com/Ruichen0424/DSN

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The Dormant Spiking Neuron (DSN) model introduces a state-driven mechanism to address computational overhead in spiking neural networks by allowing neurons to enter a dormant state when their

Key concepts

Dormancy Gate Update: This mechanism dictates when a neuron becomes inactive. It updates the activity state based on the previous membrane potential. If this potential falls below a negative threshold, the gate irreversibly closes, putting the neuron into a dormant state where it no longer requires computation.
Surrogate Gradients (SG): Since standard training methods struggle with non-differentiable functions like the Heaviside step function used for dormancy, surrogate gradients are employed. This technique replaces the derivative of that function with a smooth approximation during backpropagation, allowing the network to learn how to set these dormancy constraints effectively.
Multiplicative Reduction in Load: The DSN achieves efficiency gains by combining two types of sparsity: input-driven and state-driven. By allowing neurons to enter a dormant state based on their internal potential, the model prunes computations orthogonally to standard input pruning, leading to a multiplicative reduction in the overall computational load.

Terminology

Spiking neural networks — 1× in this paper · explained in 2 episodes

Summary

The Dormant Spiking Neuron (DSN) model introduces a state-driven mechanism to address computational overhead in spiking neural networks by allowing neurons to enter a dormant state when their membrane potential falls below a negative threshold, thereby dynamically pruning redundant computations at the instance level.

The gist: The DSN framework consistently reduces computational energy by 40% to over 70% while maintaining competitive performance on static and neuromorphic datasets, outperforming state-of-the-art static pruning techniques on ImageNet by simultaneously improving baseline SNN accuracy by 1.34%

The Problem Addressed

A substantial yet chronically overlooked source of computational overhead persists within the SNN framework, stemming from the continuous subthreshold dynamics of postsynaptic neurons <ref:2610.21748#pg3>. Quantitative analysis reveals that up to 44% of neurons in an ImageNet-trained ResNet-18 model reside in a deeply inhibited state, imposing a persistent computational burden despite having a negligible probability of firing <ref:2610.21748#pg3>. This inefficiency is quantified by observing that while approximately 26% of neurons are in the active spiking region and 30% reside in the subthreshold active region, 44% of the neurons are found in the deeply inhibited region (Vm ≤ 0) <ref:2610.21748#pg3>.

The DSN Mechanism

The core innovation is a state-driven dynamic gate, denoted as Dt n ∈ (0, 1), which allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold <ref:2610.21748#pg4>. This mechanism operates orthogonally to conventional presynaptic input-driven sparsity, enabling multiplicative reduction in overall computational load <ref:2610.21748#pg4>. The complete forward dynamics follow four stages at each timestep t, including the Dormancy Gate Update where Dt n = Dt−1 n · Θ(Vt−1 n − Dth) <ref:2610.21748#pg4>. Crucially, a dormant neuron will never fire again, meaning subsequent dynamics for that neuron need not be executed <ref:2610.21748#pg5>.

Training via Surrogate Gradients

To enable training despite the non-differentiable nature of the Heaviside step function Θ(·), the DSN utilizes backpropagation through time (BPTT) employing surrogate gradients (SG) <ref:2610.21748#pg5>. This involves using the exact function in the forward pass and substituting its derivative with a continuous approximation, σ′(·), during the backward pass <ref:2610.21748#pg5>. The recurrence relations for error terms are defined by equations such as δt H,n = δt S,n · Dt n · σ′s(Ht n − Vth) + δt V,n · (1 − S t n) <ref:2610.21748#pg14>. This allows the network to learn a robust dormancy policy that preserves critical information flow <ref:2610.21748#pg3>.

Computational Efficiency Gains

The DSN achieves multiplicative reduction in computational cost by combining input-driven and state-driven sparsity <ref:2610.21748#pg6>. The SOP count for a DSN layer n is reduced to SOPsDSN,n = MACsn · X T t=1 (Rt n−1 · Dt n) <ref:2610.21748#pg17>. This demonstrates that the DSN converts a large portion of the identified deeply inhibited population into the dormant state, addressing computational redundancy <ref:2610.21748#pg10>. The Neuronal Computational Utilization (NCU) metric measures this, and empirical results suggest that the NCU surpasses 70% across most network layers <ref:2610.21748#pg10>.

Performance and Robustness

Extensive experiments demonstrate that the DSN framework consistently reduces computational energy by 40% to over 70% across static and neuromorphic datasets <ref:2610.21748#pg7>. On ImageNet, the DSN outperforms state-of-the-art static pruning techniques at comparable reported cost reductions while simultaneously improving the corresponding baseline SNN accuracy by 1.34% <ref:2610.21748#pg3>. The analysis of the dormancy threshold Dth indicates that for static datasets, Dth = −0.05 remains near the low-energy plateau <ref:2610.21748#pg9>. Furthermore, the Jaccard similarity of dormant masks shows high heterogeneity across inputs, supporting the conclusion that the DSN leverages instance-level rather than dataset-level redundancy <ref:2610.21748#pg19>.

Training and Inference Paradigms

The DSN framework offers two paradigms for application: inference-time optimization (ITO) and direct training via BPTT <ref:2610.21748#pg7>. The ITO paradigm acts as a plug-and-play optimization tool applied after training, while the BPTT paradigm incorporates the dormancy mechanism directly into end-to-end training to learn a robust policy <ref:2610.21748#pg7>. The DSN model is compatible with rate coding and multi-spike architectures, but it may be less suitable for Time-to-First-Spike (TTFS) coding due to limited informational redundancy <ref:2610.21748#pg10>.

Model Parameters

Throughout all experiments, the networks utilize the Leaky Integrate-and-Fire (LIF) neuron model with a membrane time constant set to τ = 2.0, a firing threshold of Vth = 1.0, and a hard reset potential of Vreset = 0.0 <ref:2610.21748#pg9>. The surrogate function σ′(x) is derived from the sigmoid function σ(x) = 1 / (1 + e−αx), with the sharpness parameter set to α = 4.0 <ref:2610.21748#pg9>. This smooth approximation provides stable gradient flow for learning robust dormancy policies <ref:2610.21748#pg9>.

Conclusion

The DSN integrates a mechanism that allows neurons to enter a permanent dormant state based on their membrane potentials, thereby bypassing redundant computations at the instance level <ref:2610.21748#pg11>. The DSN framework offers an effective approach for improving neuromorphic computing efficiency by providing a trainable, state-driven sparsity mechanism orthogonal to conventional presynaptic input-driven sparsity <ref:2610.21748#pg11>. This work addresses the substantial but often overlooked computational overhead in SNNs arising from continuous subthreshold updates of deeply inhibited neurons <ref:2610.21748#pg11>. The DSN framework reduces estimated energy consumption by 40% to over 70% across static and neuromorphic datasets and network architectures while maintaining competitive task accuracy relative to the baselines <ref:2610.21748#pg11>. This work addresses the substantial but often overlooked computational overhead in SNNs arising from continuous subthreshold updates of deeply inhibited neurons <ref:2610.21748#pg11>. The DSN framework reduces estimated energy consumption by 40% to over 70% across static and neuromorphic datasets and network architectures while maintaining competitive task accuracy relative to the baselines <ref:2610.21748#pg11>. This work addresses the substantial but often overlooked computational overhead in SNNs arising from continuous subthreshold updates of deeply inhibited neurons <ref:2610.21748#pg11>. The DSN framework reduces estimated energy consumption by 40% to over 70% across static and neuromorphic datasets and network architectures while maintaining competitive task accuracy relative to the baselines <ref:2610.21748#pg11>.

--- Page 26 ---

How it works

The DSN incorporates two primary modifications compared to conventional spiking neurons: the dormancy gate update mechanism specified in eq. (1) and the gating of the spike output by Dt n in eq. (3) <ref:2610.21748#pg4>. The activity state Dt n is updated based on Vt-1 n, and if Vt-1 n drops below Dth, the gate irreversibly closes (Dt n → 0), making the neuron dormant <ref:2610.21748#pg4>. This property underlies the computational savings provided by the DSN <ref:2610.21748#pg5>.

Training via Surrogate Gradients

The BPTT training methodology is formulated using surrogate gradients (SG) to mitigate performance degradation from non-differentiable functions like Θ(·) <ref:2610.21748#pg5>. The core principle involves using the exact function in the forward pass but substituting its derivative with a continuous approximation, σ′(·), during the backward pass <ref:2610.21748#pg5>. This enables the network to adapt its weight distribution to the dormancy constraint and learn a robust dormancy policy <ref:2610.21748#pg3>.

Computational Efficiency Gains

The DSN model combines input-driven sparsity and state-driven sparsity, yielding a multiplicative reduction in computational cost as SOPsDSN,n = MACsn · X T t=1 (Rt n−1 · Dt n) <ref:2610.21748#pg6>. This yields a computational cost lower than that of either sparsity type alone <ref:2610.21748#pg6>. The efficiency benefit provided by the DSN increases with the number of timesteps, growing to over 63% at T = 16 and exceeding 73% by T = 96 <ref:2610.21748#pg9>.

Performance and Robustness

The DSN consistently reduces estimated energy consumption by 40% to over 70% across static and neuromorphic datasets <ref:2610.21748#pg11>. The results also suggest an interaction between the DSN mechanism and transformer architectures, with applying it to the SD-Transformer consistently yielding substantial energy savings with limited impact on accuracy <ref:2610.21748#pg7>. The Jaccard similarity of dormant masks shows high heterogeneity across inputs, supporting the conclusion that the DSN leverages dynamic, instance-level transient redundancy <ref:2610.21748#pg19>.

Improvements for AI systems

textbfState-Driven Sparsity for Computational Pruning: The DSN Model Implementation in Inference Time Optimization (ITO). You can improve AI systems by implementing a state-driven sparsity paradigm that specifically targets the computational inefficiency of deeply inhibited neurons, which is achieved by introducing a state-driven dynamic gate, the DSN allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold. This enables multiplicative reduction in overall computational load by operating orthogonally to conventional presynaptic input-driven sparsity.
textbfEnd-to-End Learning of Robust Dormancy Policies: Training via Backpropagation Through Time (BPTT) with Surrogate Gradients. The system can be trained to adapt its structure by using the DSN's mechanism directly integrated into the neuronal dynamics and trained end-to-end via backpropagation through time using surrogate gradients, allowing it to learn a robust dormancy policy that preserves critical information flow.
textbfIncreased Computational Efficiency: Achieving Significant Energy Reductions. The improved system can achieve substantial efficiency gains by demonstrating that the DSN framework consistently reduces computational energy by 40% to over 70% across static and neuromorphic datasets, and on ImageNet, it outperforms state-of-the-art static pruning techniques at comparable reported cost reductions.
textbfAdaptive Pruning for Temporal Data: Handling Nonstationary Inputs. The system can be made robust for dynamic inputs because the DSN leverages state-driven sparsity to target instance-level, transient redundancy, addressing the limitation of static pruning where early inhibition does not necessarily imply future unimportance in continuous or nonstationary temporal streams.
textbf Enhanced Feature Preservation: Maintaining Accuracy During Sparsification. The system can maintain high performance while pruning by showing that on ImageNet, the DSN achieves comparable reported cost reductions and simultaneously improves the corresponding baseline SNN accuracy by 1.34%.

Listen

Transcript

Key concepts

Terminology

Summary

Improvements for AI systems

Related papers