Phase transitions reveal hierarchical structure in deep neural networks

The gist

Phase transitions in DNN learning are governed by saddle points in the loss landscape, revealing a hierarchical structure of accuracy basins analogous to phases in statistical physics.

Geometric Origin of Phase Transitions

Analytical work shows that phase transitions in DNN learning are universally governed by critical saddle points on the loss landscape that are associated with concave basin boundaries on the underlying un-regularized error landscape. The addition of L2 regularization shifts all eigenvalues upward, turning a direction with a negative eigenvalue on the error landscape into a saddle point on the loss landscape at the right beta. This mechanism provides a first-principles explanation for the hierarchical learning process in deep neural networks.

Hierarchical Structure and Phase Transitions

The work establishes that each transition is locally governed by the emergence of a saddle point, created when the L2 term amplifies a concave region of the underlying unregularized error landscape. The landscape is organized into concentric, hierarchically nested phases defined by weight norm. These phases are intrinsically connected by low-error, flat pathways. The transitions are accompanied by measurable geometric signatures in the error landscape’s scalar curvature.

Probing Landscape Geometry with Pathfinder

The authors introduce the Pathfinder algorithm to explore the geometry of error landscapes, defined as Lβ(θ) = E(θ) + β · θ − θref2. This method allows for tracing paths through the error landscape by incrementally increasing regularization strength. The algorithm is used to identify hierarchical phase transitions between accuracy basins and find a striking correspondence between basins and specific digits on the MNIST dataset.

Mode Connectivity Confirmation

The paper confirms the existence of almost flat paths between minima on the error surface (the so-called mode connectivity). The Pathfinder algorithm was used to show that there exist paths along which the error does not change substantially. This demonstrates that the landscape is not only phase-structured but also internally connected, with smooth, flat paths between optimal models.

Statistical Physics Analogy and Practical Implications

The findings position neural networks within the realm of physical systems amenable to analysis by statistical physics concepts. The regularizer acts as a thermodynamic parameter tuning the system between phases. This mapping provides a mechanistic explanation for empirical observations, such as transfer learning and curriculum learning. The Pathfinder method is presented as a computationally cheap and practical analogue to the Information Bottleneck method.

Limitations and Future Directions

The method relies on the assumption that the error landscape possesses a degree of smoothness and connectivity that allows for meaningful paths to be traced. The numerical results are currently confined to feed forward architectures, though it is hypothesized that the underlying principle should hold for any sufficiently complex, nonconvex model space. A critical open question is how this hierarchy manifests in other architectures like CNNs or Transformers.

The work establishes the geometric origin of key phenomenology in DNN training and demonstrates the existence of hierarchically ordered accuracy basins. These basins correspond to models in non-trivial error landscapes that are strictly analogous to phases in statistical physics. The authors have established a formal geometric theory connecting the physics of phase transitions to the dynamics of deep learning. This perspective treats the learning process as a physical process exploring a structured landscape, where the shifted L2 strength beta acts as a thermodynamic parameter tuning the system between phases. By re-conceptualizing a standard regularizer as a probe, they have opened the black box of DNNs further. The statistical physics analogy allows them to understand them as systems with a comprehensible internal organization. This not only deepens their theoretical understanding of Deep Learning but also provides a practical path toward more robust, interpretable, and efficient machine learning.

How it works

1. Analytical proof shows that phase transitions are governed by critical saddle points on the loss landscape associated with concave basin boundaries on the underlying un-regularized error landscape.

2. L2 regularization transforms pre-existing concave boundaries in E(θ) into saddle points in the loss landscape at critical strengths.

3. The Pathfinder algorithm probes this geometry by incrementally increasing regularization strength and training models to minimize the loss function Lβ,θref (θ) = E(θ) + β · θ − θref2.

4. This systematic probe reveals hierarchically nested phases defined by Euclidean norm of the parameter vector.

5. The algorithm confirms mode connectivity by tracing paths where the error remains nearly constant, showing smooth, flat paths between optimal models.

Probing saddle points and concave basin boundaries using the Pathfinder

The authors empirically validated the mechanism by applying Pathfinder to an MNIST classifier. They observed five phase transitions corresponding to jumps in parameter space and linked these transitions to learning/forgetting events of specific digits. Negative curvature directions on the loss landscape emerge just before a phase transition takes place, providing direct evidence of saddle-point formation.

Probing and confirming mode connectivity using the Pathfinder

The authors randomly chose five optimal models and set the reference point θref to one model's parameters to initialize the others, then varied beta on the order of 10−6. This confirmed that models are connected by flat, low-error paths (mode connectivity).

Robustness of Trajectories

Robustness tests verified that trajectories generated with different learning rates and different endpoint configurations consistently crossed transition boundaries at the same critical Euclidean norm. This reproducibility confirms their origin in the underlying landscape geometry. The identified transitions are robust w.r.t. learning rate and L2 reference point.

Reproducibility and Data Availability

All experiments used publicly available data, such as the MNIST dataset. The full source code for model training, Pathfinder trajectory generation, and Hessian analysis is available at a public data repository. The authors will make the code and materials available in a public data repository after peer review is complete.

Conclusions and Future Directions

The work opens avenues for applying statistical physics methods to quantitatively characterize error landscapes. The shifted L2 probe (Pathfinder) provides a new diagnostic framework for analyzing model behavior and designing better training schemes. This lays the groundwork for reverse-engineering the hierarchical features a network acquires. A better understanding of the error-landscape geometry will be key for developing better strategies and methods for continuous learning, transfer learning and model merging.

--- Page 12 ---

Acknowledgements

We are grateful to Björn Ladewig for the insightful discussions. A.F.L.C. acknowledges the DAAD for scholarship Research Grants - One-Year Grants for Doctoral Candidates (7693452) to the University of Potsdam.

Author Contributions

I.T.E. and K.W. designed the study, I.T.E. conceptualized the idea of the Pathfinder. I.T.E. and A.F·L·C designed the methodology. A·F·L·C conducted the numerical experiments and produced the figures. All authors discussed the results and contributed to the writing of the manuscript.

Competing Interests

The authors declare no competing interests.

References

[1] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries, 2023 [2] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016 [3] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016 [4] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning [5] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning [6] Alan J. Bray and David S. Dean. Statistics of critical points of gaussian fields on large-dimensional spaces. Phys. Rev. Lett., 98:150201, Apr 2007 [7] Anna Choromańska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In International Conference on Artificial Intelligence and Statistics [8] George Cybenko.

1. The AI system can be trained to identify accuracy basins corresponding to specific digits in MNIST by analyzing model trajectories as a function of L2 regularization strength beta. This allows for reverse-engineering the network's internal representation by tracking which features are learned/forgotten at each transition T1 to T5.

2. The system can be designed to proactively navigate training toward desired accuracy regimes by utilizing the Pathfinder algorithm, which probes and traverses transition boundaries. This enables the AI to avoid sharp, hard-to-traverse geometric barriers identified as concave boundaries on the error landscape.

3. The system can confirm mode connectivity between independently discovered optimal models by using a shifted L2 probe to trace paths where the error does not change substantially. This allows for faster model merging and ensembling by finding low-error, flat pathways between solutions, which is computationally more efficient than existing optimization-based methods.

Training Deep Neural Networks relies on the model converging on a high-dimensional, non-convex loss landscape toward a good minimum. Yet, much of the phenomenology of training remains ill understood. We focus on three seemingly disparate observations: the occurrence of phase transitions reminiscent of statistical physics, the ubiquity of saddle points, and phenomenon of mode connectivity relevant for model merging. We unify these within a single explanatory framework, the geometry of the loss and error landscapes. We analytically show that phase transitions in DNN learning are governed by saddle points in the loss landscape. Building on this insight, we introduce a simple, fast, and easy to implement algorithm that uses the L2 regularizer as a tool to probe the geometry of error landscapes. We apply it to confirm mode connectivity in DNNs trained on the MNIST dataset by efficiently finding paths that connect global minima. We then show numerically that saddle points induce transitions between models that encode distinct digit classes. Our work establishes the geometric origin of key training phenomena in DNNs and reveals a hierarchy of accuracy basins analogous to phases in statistical physics.