From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

As a fastidious and diligent AI researcher, I have meticulously reviewed both provided texts concerning the paper From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes. The synthesis below integrates the core theoretical mechanisms from Text A (the primary research description) with the rigorous mathematical derivations and classification frameworks presented in Text B (the detailed technical findings).

This combined summary aims to provide a comprehensive, high-fidelity understanding of the paper's contributions, focusing on the interplay between spectral dynamics, learning schedules, noise accumulation, and resulting risk asymptotics.

Comprehensive Research Synthesis: Spectral Dynamics and Schedule Dependence in LLM Pre-training Loss Landscapes

This research investigates the complex dynamics governing power-law learning curves in noisy online Stochastic Gradient Descent (SGD) applied to models characterized by linear random features. The core thesis is that the observed loss trajectory is not solely determined by fixed model properties or data characteristics, but rather emerges from a dynamical response jointly shaped by the spectral structure, target alignment, intrinsic schedule parameters, and accumulated stochastic noise.

I. The Theoretical Framework: Forcing-Memory Recursion

The foundation of this analysis rests on modeling the prediction risk via a Volterra equation, which decomposes the loss into two fundamental components:

1. The Forcing Term (F): This component propagates unresolved target error, representing the clean decay dictated by the underlying model and data structure, without stochastic feedback.

2. The Memory Kernel (K): This component describes how long the effects of discrete stochastic-error injections (noise) persist in the system's response.

This leads to a two-stage mechanism:

Spectral Origin to (F, K) forcing R sigma,t observed loss

The crucial insight is that the power-law behavior of either component (F or K) is not guaranteed by coordinatewise power laws for individual eigenvalues or target coefficients. Instead, the criterion for a component to exhibit a power law is dictated by its cumulative weighted spectral mass near zero. This spectral origin dictates whether the forcing term or the memory kernel adheres to low-spectrum scaling.

II. The Canonical Model and Scaling Regimes

The interaction between these components under different scheduling regimes defines the system's behavior, leading to a specific classification of dynamics:

• Intrinsic Time (T t): This coordinate, T t = P s<t eta s, controls the overall optimization progress dictated by the learning-rate and batch-size schedule.

• Noise Injection Rate (r t): This is controlled by the ratio of batch size to learning rate, r t = B/ eta t.

The interplay between these two natural schedule coordinates yields a 3 + 3(+2) map, which precisely characterizes the system's behavior across different propagation subregimes:

• 3 Long-Memory (LM) regimes: Characterized by long-range dependencies.

• 3 Integrable-Memory (IM) regimes: Characterized by finite, integrable memory effects.

• 2 Finite-Bulk (FB) propagation regimes: Representing the behavior in the bulk of the spectrum.

These subregimes are further refined into specific scaling laws, such as exact conditional finite-bulk gap laws, providing deterministic equivalents for risk under these conditions.

III. Schedule Transformation and Noise Accumulation

A major contribution is understanding how joint learning-rate/batch-size schedules transform the clean power law:

• Noise Accumulation: The observed noisy–clean gap is explicitly shown to be obtained by accumulating schedule-dependent stochastic injections through the memory kernel K.

• Schedule Classification: This mechanism establishes sharp boundaries between different schedules, classifying them into three categories: preserve, change, or destroy the clean power law.

• Memory Ceiling: A critical finding is the existence of a memory ceiling on noise reduction, beyond which merely reducing late-stage noise is insufficient to improve the decay rate.

IV. Empirical Validation and LLM Application

The theoretical framework is rigorously tested using controlled experiments on nanoGPT models:

1. Coordinate Characterization: Experiments confirm that the loss can be fully characterized by the theory-derived coordinates: intrinsic time (T t) and the noise injection ratio path (r t = B/eta t).

2. Surrogate Prediction: A forcing-memory surrogate model, fitted on one schedule, accurately predicts held-out losses across different schedules without requiring refitting. This strongly supports a response-level mechanism beyond frozen features.

3. Effective Exponents: The analysis links the theory to empirical observations; for instance, the fitted memory exponent q K about 1 corresponds to an effective alpha about 1/2, which aligns with theoretical predictions (from the 4+3 theory) for near-square-root compute-optimal width scaling reported in literature (e.g., Chinchilla et al., Hoffmann et al.).

V. Mathematical Rigor and Resource Optimization

Text B provides the necessary mathematical depth, detailing how these concepts translate into rigorous risk asymptotics:

• Risk Asymptotics: The analysis derives bounds on the total centered loss (R sigma(T) - R app) across various branches (FB1, FB2).

• Preserve–Change–Destroy Classification: Theorem D.21 formally classifies the behavior of the noisy–clean gap exponent based on the relationship between the clean risk exponent (q 0) and a schedule-dependent noise exponent (q N).

• Optimal Control: The paper provides five detailed steps to derive optimal schedule families that attain optimal data and feature-compute rates, including deriving the required ratio profile r(T) for specific asymptotic equivalences.

• Compute Rate Analysis: Appendix E.4 summarizes the optimal compute rates derived under fixed noise variance (sigma squared) across different propagation subregimes, showing how compute shifts from iteration count to model width as noise levels change.

Conclusion

The paper fundamentally redefines the understanding of power-law learning curves by moving beyond component-wise analysis. It establishes that observed loss is a dynamical response governed by the competition between clean target-error decay and accumulated stochastic response, mediated by spectral mass, target alignment, and schedule control. The resulting 3 + 3(+2) map provides a precise taxonomy for LLM pre-training dynamics, allowing researchers to predict how learning schedules will affect loss trajectories and determine the resource-optimal trade-offs between model width and training compute budget under realistic noise conditions.

1. The improved system can precisely characterize power-law learning curves by determining if either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling, distinguishing between forcing terms (driven by target error) and memory kernels (driven by stochastic error injection).

2. The system can predict held-out schedules without refitting by utilizing a forcing-memory surrogate fitted on one schedule (e.g., 8-1-1) predicts held-out WSD trajectory without refitting, which is validated across OpenWebText and FineWeb datasets, demonstrating that the LLM response coordinates are dataset-robust.

3. The system can optimize resource allocation by identifying sharp conditions under which a schedule preserves, changes, or destroys the clean power law, providing a phase classification via the 3 + 3(+2) map to determine whether it is in a Long Memory (LM), Integrable Memory (IM), or Finite Bulk (FB) regime.

4. The system can design optimal training schedules by calculating optimal ratios and sharp resource rates based on data budget and feature-compute budgets, yielding specific R⋆σ(D) R⋆σ(f) orders in Tables 1 and 2 for various regimes.

5. The system can adapt to varying optimization strategies by using the joint learning-rate/batch-size schedules framework, where the observed loss is determined by the accumulation of schedule-dependent stochastic injections through a memory kernel, providing sharp boundaries between schedules that preserve, change, or destroy the clean power law.

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time T t= sum s<tη s controls optimization progress, while r t=B t/η t controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in 3+3(+2) propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched B/η paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in 3+3(+2) map. The source code for reproducing our experiments is available on https://github.com/yichenblue/spectra-to-schedules-in-pretraining.