Toward a Physics of Deep Learning and Brains

The gist Deep neural networks and brains both learn and share superficial similarities, allowing for a unified theoretical framework derived from non-equilibrium statistical physics to describe their activity. This framework shows that deep neural networks learn best when poised between absorbing and active phases, suggesting that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself.

Theoretical Foundation and Criticality

The paper establishes that neuronal avalanches can be applied to cascades in deep neural networks, showing that these equations derived from non-equilibrium statistical physics demonstrate learning occurs best when networks are poised between absorbing and active phases

The concept of neuronal avalanches provides an account matching scale-free statistics observed in neuronal data, with dynamic fluctuations typical of critical phase transitions. The original framework required external inputs to be small and rarely occur, leading to the concept of quasi-criticality which states that as neuronal populations are driven, they hover within a tunable neighborhood of a critical point. Current studies verify proximity to criticality using crackling noise theory through tests like the power law distribution of avalanche sizes P(S) ∼ S−τs and durations P(D) ∼ D−τd.

Characterizing Criticality in Deep Networks

The paper performs event-resolved, crackling-noise analysis for deep learning and finds that Gaussian initialised networks exhibit signatures of a genuine non-equilibrium phase transition belonging to the Barkhausen universality class. For ResNets, they also exhibit signatures of criticality with exponent relations consistent with mean-field directed percolation.

The paper distinguishes between two types of criticality: the edge of chaos and the Widom-like line. The Widom-like line correlates with learning performance loss as it fades as σ2b grows.

Link to Learning Performance

The analysis links computation to dynamics by showing that trainability overlaps a quasi-critical plateau, noting that the control parameters giving heightened susceptibility also enable learning. Empirical results show that successful training overlaps with the area of heightened susceptibility, and as σ2b increases, the trainable region narrows and shifts to larger values of σ2w

The paper concludes that learning tasks require certain levels of proximity to criticality, as trainability and learnability starts to be achieved for those networks initialized near the critical value σ2w ≈ 1.

Broader Implications

The crackling toolkit provides operational diagnostics for training by allowing pipelines to track exponent-relation mismatch and distance from criticality. Standard knobs like dropout and batch normalization can serve as advanced control parameters to bias training toward the quasi-critical plateau. The study predicts that task specific performance can vary between different criticalities characterized by sets of exponents.

The work demonstrates that universal features are shared by both biological and artificial neural networks through distinct universality classes like Barkhausen noise and directed percolation. This theoretical framework provides a blueprint for engineering improved network performance. The commonalities between brains and deep neural networks lead both systems to share physical laws describing maximal susceptibility along the Widom line.

Methods Summary

Mean-field theory shows that at the critical point (σ2w, σ2b) = (1, 0), the steady state signal strength q(MF)ss can take the nontrivial branch q(MF)ss = 1/2 − 1/σ2w. The mean-field directed percolation universality class exponents β = 1 and σ = 2 reflect the same critical behavior. Avalanche characterization involves tracking signal strength √ql evolution relative to an input threshold √q0

The study also explores highly engineered structures like ResNets and finds that they too exhibit signatures of criticality with exponents consistent with mean-field directed percolation

The paper provides an event-resolved, non-equilibrium phase transition framework to understand deep learning by resolving propagation into avalanches The findings suggest that learning is supported not by exact point criticality but by quasi-critical plateaus. The final conclusion is that both brains and deep neural networks use avalanches or cascades of activity to transmit information through stages or layers of processing units.

The paper is 2509.

1. Weight Initialization Strategy: Instead of relying on proximity to a critical point, systems should be initialized such that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This allows for engineering improved network performance by targeting the region where trainability overlaps a quasi-critical plateau.

2. Training Regimes Control: To steer models toward optimal learning, researchers can utilize control parameters like dropout or batch normalization as more advanced control parameters for criticality—for instance, in terms of their effect on the susceptibility. This enables regularizers to bias training toward the quasi-critical plateau where performance peaks.

3. Architecture Design for Specific Tasks: For highly engineered architectures like ResNets, the system can be designed to exhibit signatures of criticality by tuning layer-dependent thresholds: assigning layer dependent thresholds θl = µl + nσl, where n is an integer and µ and σ are mean and standard deviation of √ql. This allows for the study of distinct universality classes like mean-field directed percolation.

4. Diagnostic Tool Development: Implement an event-resolved analysis pipeline to provide operational diagnostics for training, tracking the exponent-relation mismatch, collapse quality, and, ultimately, distance from criticality. This allows practitioners to locate and steer models within the quasi-critical regimes.

Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.