Learning reshapes power-law anisotropy in internal representations

arXiv:2608.15239 · cs.LG, stat.ML · Submitted 2026-08-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Learning reshapes power-law anisotropy in internal representations".

Jane: Here's the gist of the paper. The gist: The internal-representation spectrum in neural networks is dynamically reshaped by the interaction between input statistics and task learning, exhibiting up to four distinct asymptotic regimes during feature learning.

Tom: Characterization of Internal Representation Geometry Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex This anisotropy is a key geometric property of high-dimensional information processing and underlies a variety of theoretical analyses The eigenspectrum of the covariance matrix, or kernel, of the internal representations is a fundamental quantity characterizing this geometry Recent studies have reported that the eigenvalues of internal-representation covariance matrices decay as a power law with rank in a wide range of neural systems Such power-law anisotropy implies that a large fraction of the variance is concentrated along a small number of directions, while information remains distributed across many higher-order directions Mechanism in Linear Neural Networks The study characterizes this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher–student setting with power-law input and teacher structures In this setting, the internal-representation kernel is defined as Φt(x, x′) = x⊤Gw(t)x′, where Gw(t) is the Gram matrix of the first-layer weights The temporal evolution of the eigenvalues of the internal-representation kernel is given by ρk(t) = σ 2 wλk X + λk∆ηk(t) X, where ∆ηk(t) is the teacher-derived component formed through learning Asymptotic Regimes of the Local Power-Law Exponent The local effective power-law exponent, αeff(k, t), is defined as −∂ log ρk(t) / ∂ log k In the feature-learning regime, the local effective power-law exponent can attain as many as four distinct asymptotic values over finite ranges of modes as learning progresses These four regimes are: 1.

Jane: α, Rk ≪ 1, where the initialization component dominates 2.

Tom: 3α + β, Rk ≫ 1, when learning-induced change exceeds the initialization component 3.

Title and authors: Tom: Let's talk about the title and who wrote this, "Learning reshapes power-law anisotropy in internal representations". It immediately tells you that the learning process itself is actively changing the geometry of the internal representation, specifically something called power-law anisotropy.

Jane: So, if I’m driving and I see a road that looks like one thing initially but then I learn about traffic patterns and things change, this paper suggests my brain's internal map doesn't just get better; it gets reshaped in a structured way related to those power laws.

Lu: The authors are using this two-layer linear neural network setup with power-law input and teacher structures to exactly characterize how this happens. They’re trying to figure out if the representations simply inherit the input structure or if they are actively shaped by the task learning.

Meng: I wonder what that means for scaling. If a model’s internal structure is constantly changing its geometric properties, maintaining stability across different tasks might be really hard to manage at scale.

Lalam: The paper shows that in the feature-learning regime, the local power-law exponent of the internal-representation spectrum evolves nonmonotonically over the course of training and exhibits up to four distinct asymptotic regimes across modes and training times.

The paper's summary: Tom: So, what does that mean for us? They show that during feature learning, this local effective power-law exponent can hit up to four different values depending on the mode index and how long you’ve been training. That’s a lot of distinct geometric behaviors happening at once.

Jane: Four regimes sounds complicated for someone who isn't reading the math, but it suggests that what the network focuses on changes dramatically as it learns more. It implies that different parts of the network are prioritizing different types of information in sequence.

Lu: They define these four regimes based on whether the initialization component dominates or if learning-induced change exceeds that initialization component, and they tie this to how much learning happens in the first layer versus both layers.

Meng: If we're training a massive model, we need to know which regime we’re in so we can predict how the feature space will evolve. That predictive power is what I care about from an engineering view.

Lalam: In contrast, they show that in the lazy regime, where learning-induced change never exceeds the initialization component even for the strongest teacher mode, the exponent remains essentially unchanged and equals sigma squared w lambda k.

The paper's improvements: Tom: The authors point out that these four distinct regimes only exist under specific hyperparameter conditions, requiring things like sigma a being much larger than sigma w and gamma zero being much larger than the sum of sigma squared a and sigma squared w theta zero for the feature-learning regime.

Jane: So, it’s not just that learning happens; it depends heavily on how strong the input statistics are compared to the network's weights and learning rates. It’s a delicate balance.

Lu: They also show that in more realistic nonlinear networks with a nonlinear activation function, you still see similar exponent dynamics—the spectrum changes successively from low to high mode index, and its exponent evolves from alpha to alpha plus beta over a finite range of modes during feature learning.

Meng: That’s interesting because it means we don't just need perfect linear models to see this; the mechanism seems more fundamental than just the specific architecture we use. That broad applicability is important for deployment.

Lalam: The conclusion they draw is that power-law internal representations shouldn't be seen as having a single fixed exponent; instead, their exponents are reshaped over time and across modes through the interaction between input statistics and task learning, which provides an analytical starting point for understanding them.

Conclusion: Tom: So to wrap up, the main thing from "Learning reshapes power-law anisotropy in internal representations" is that the geometry of a neural network's internal state isn't static; it’s constantly being reshaped by how much task learning happens compared to the initial structure.

Jane: It suggests that if we want to understand why some AI models are good at certain things and others aren't, we have to look at this interaction between the input data statistics and the specific learning process.

Lu: The paper provides an analytical starting point for understanding how power-law representations are inherited and transformed in neural networks by looking at the modal structure of the teacher signal and mode-dependent learning timescales.

Meng: For us in engineering, it means that predicting spectral evolution could allow us to proactively adapt our training strategies based on which regime we are entering during deployment.

Lalam: I just want to add that they rely on simultaneous diagonalizability of the input covariance and the teacher matrix, infinite width, and population-limit gradient flow for their analysis, which is a big assumption they made in their model.

Asahi Nakamuta, Jun-nosuke Teramae

Graduate School of Informatics, Kyoto University

cs.LG, stat.ML

Submitted: 2026-08-15

Updated: 2026-10-21

Comments: 29 pages, 6 figures

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist: The internal-representation spectrum in neural networks is dynamically reshaped by the interaction between input statistics and task learning, exhibiting up to four distinct asymptotic

Key concepts

Power-law Anisotropy: This describes a geometric property of high-dimensional data where most of the variance in internal representations is concentrated along a few dominant directions, while information is spread across many other directions. It's a key feature observed in biological and artificial neural systems.
Internal Representation Kernel: This mathematical object represents the relationships between different input samples within the network's internal state. Its eigenvalues characterize the geometry of these representations, showing how variance is distributed across different modes during learning.
Local Effective Power-Law Exponent: This exponent measures how quickly a specific mode's power-law decay changes as training progresses. It quantifies whether a particular representation mode is being dominated by initial setup or actively shaped by the task learning process.
Asymptotic Regimes: These are distinct, stable states the local power-law exponent can reach during feature learning. The paper identifies four such regimes, indicating different ways the internal representation structure adapts depending on whether initialization or task learning dominates.

Terminology

Anisotropy — 1× in this paper · explained in 4 episodes

Summary

The gist: The internal-representation spectrum in neural networks is dynamically reshaped by the interaction between input statistics and task learning, exhibiting up to four distinct asymptotic regimes during feature learning.

Characterization of Internal Representation Geometry

Mechanism in Linear Neural Networks

The study characterizes this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher–student setting with power-law input and teacher structures In this setting, the internal-representation kernel is defined as Φt(x, x′) = x⊤Gw(t)x′, where Gw(t) is the Gram matrix of the first-layer weights The temporal evolution of the eigenvalues of the internal-representation kernel is given by ρk(t) = σ 2 wλk X + λk∆ηk(t) X, where ∆ηk(t) is the teacher-derived component formed through learning

Asymptotic Regimes of the Local Power-Law Exponent

The local effective power-law exponent, αeff(k, t), is defined as −∂ log ρk(t) / ∂ log k In the feature-learning regime, the local effective power-law exponent can attain as many as four distinct asymptotic values over finite ranges of modes as learning progresses These four regimes are:

α, Rk ≪ 1, where the initialization component dominates
3α + β, Rk ≫ 1, when learning-induced change exceeds the initialization component
α + β, Rk ≫ 1, when learning is primarily in the first layer
α + β/2, Rk ≫ 1, when learning occurs in both layers

Regime Dependence and Generalization

The existence of all four regions depends on the hyperparameters, specifically requiring σa >> σw and γ0 >> σ 2 a + σ 2 w θ0 for the feature-learning regime In contrast, in the lazy regime satisfying γ0 << pσ 2 w(2σ 2 w + σ 2 a)θ0, the training-induced component never exceeds the initialization component even for the strongest teacher mode, and ρk(t) ≃ σ 2 wλk, αeff(k, t) ≃ α Numerical experiments with nonlinear networks confirm that these predicted spectral transitions are observed beyond the linear model

Nonlinear Network Confirmation

Numerical studies using nonlinear networks with a nonlinear activation function in a teacher–student setting show that the slope of the internal-representation spectrum also changes over the course of learning in the nonlinear network In particular, in the feature-learning regime, the spectrum changes successively from low to high mode index, and its exponent evolves from α to α + β over a finite range of modes This result suggests that the mechanism analyzed exactly in the two-layer linear model is not peculiar to linear networks, but may capture a basic mechanism of internal-representation formation in a broader class of neural networks

Conclusion on Representation Dynamics

The results show that power-law internal representations should not be regarded as possessing a single fixed exponent; rather, their exponents are reshaped over time and across modes through the interaction between input statistics and task learning The analysis provides a foundation for analytically understanding how power-law representations are inherited and transformed in neural networks The paper relies on simultaneous diagonalizability of the input covariance and the teacher matrix, infinite width, and population-limit gradient flow Future work should extend the analysis to misaligned input and teacher structure, finite-width and finite-sample effects, deeper networks, and general nonlinear activation functions The paper demonstrates that power-law internal representations are dynamically reshaped by the anisotropy of the teacher signal and by mode-dependent learning timescales The analysis provides an analytical starting point for understanding the power-law internal representations observed across neural networks as a consequence of the interaction between input structure and learning. The results suggest that the interaction among input anisotropy, the modal structure of the teacher signal, and mode-dependent learning rates may also govern internal-representation formation in more general networks.

--- Page 1 ---

LEARNING RESHAPES POWER-LAW ANISOTROPY IN INTERNAL REPRESENTATIONS Asahi Nakamuta, Graduate School of Informatics, Kyoto University, nakamuta.asahi.56z@st.kyoto-u.ac.jp Jun-nosuke Teramae, Graduate School of Informatics, Kyoto University, teramae@acs.i.kyoto-u.ac.jp ABSTRACT Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex This anisotropy is a key geometric property of high-dimensional information processing and underlies a variety of theoretical analyses However, the mechanism by which it emerges from input structure and task-driven learning has remained unclear Here, we characterize this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher–student setting with power-law input and teacher structures We show that, in the feature-learning regime, the local power-law exponent of the internal-representation spectrum evolves nonmonotonically over the course of training and exhibits up to four distinct asymptotic regimes across modes and training times By contrast, in the lazy regime, the exponent remains essentially unchanged We further demonstrate numerically that similar exponent dynamics arise in more realistic nonlinear networks Together, these results suggest a general mechanism by which the dynamic interaction between input statistics and task structure gives rise to power-law internal representations.

Improvements for AI systems

Adaptive Feature Extraction Based on Learning Regime: The improved system will dynamically adjust its internal representation focus based on whether it is in a feature-learning regime or a lazy regime, as the paper shows that in the feature-learning regime, the local effective power-law exponent can attain as many as four distinct asymptotic values.
Mode-Specific Learning Rate Allocation: The system can be designed to allocate learning across layers based on mode index, specifically differentiating between regions where learning occurs primarily in the first layer and regions where both layers learn, leading to tailored feature refinement for different data components.
Predictive Spectral Evolution: By monitoring the evolution of the local power-law exponent, the system can anticipate when a specific internal representation mode will transition from being dominated by initialization to being shaped by task learning, allowing for proactive adaptation during training.

Abstract

Power-law anisotropy in internal representations has been observed across a wide range of biological and artificial neural systems, from state-of-the-art language models to the mouse cerebral cortex. This anisotropy is a key geometric property of high-dimensional information processing and underlies a variety of theoretical analyses. However, the mechanism by which it emerges from input structure and task-driven learning has remained unclear. Here, we characterize this formation process by exactly solving the learning dynamics of a wide two-layer linear neural network in a teacher--student setting with power-law input and teacher structures. We show that, in the feature-learning regime, the local power-law exponent of the internal-representation spectrum evolves nonmonotonically over the course of training and exhibits up to four distinct asymptotic regimes across modes and training times. By contrast, in the lazy regime, the exponent remains essentially unchanged. We further demonstrate numerically that similar exponent dynamics arise in more realistic nonlinear networks. Together, these results suggest a general mechanism by which the dynamic interaction between input statistics and task structure gives rise to power-law internal representations.