Optimal L2 Regularization in High-dimensional Continual Linear Regression

arXiv:2601.13844 · cs.LG · Submitted 2026-01-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Optimal L2 Regularization in High-dimensional Continual Linear Regression".

Jane: Here's the gist of the paper. The gist The expected generalization loss in an overparameterized continual linear regression setting, when analyzed under high-dimensional random design with multiple teachers and label noise, can be expressed as a closed-form expression that reveals an optimal regularization strength scaling nearly linearly with the number of tasks.

Tom: Setting and Model The study focuses on continual linear regression where a learner is exposed to a sequence of T tasks, updating the predictor by fitting only the current task without access to previous samples.

Jane: The regularization scheme analyzed is L2 regularization, defined as wt ← argminw n∥Xtw − yt∥2 + λd∥w − wt−1∥2.

Tom: This analysis accommodates a setting broader than prior work by allowing both label noise and multiple teachers.

Title and authors: Tom: So to wrap up, this paper is about taking continual linear regression—which is learning new things without forgetting old stuff—and figuring out the perfect way to use L2 regularization, that lambda parameter.

Jane: Right, so it boils down to finding a scaling law for that lambda that works well when you have a lot of tasks T and a high-dimensional setup.

Lu: The authors prove there's this closed-form expression for the expected generalization loss after learning T tasks, and they show how it breaks down into these four components: label noise, teacher variability, interaction terms, and temporal correlations > Meng That decomposition is helpful because it shows us exactly what kind of error we're fighting when you tune that regularization strength.

Tom: Exactly. And the big takeaway is this optimal regularization strength lambda star scales nearly linearly with the number of tasks T in a lot of cases.

Jane: They give us this specific formula for iid teachers and non-zero mean teacher w star, which looks like lambda star is proportional to T over natural log T.

Lu: That scaling law is key because it gives a concrete rule for how much penalty you need to apply based on how many tasks you plan on learning.

Tom: That means we move away from guessing lambda and towards designing a system where the regularization strength adapts based on the task horizon T.

Jane: And they validate this scaling law not just with linear regression, but with neural networks too, which is important because it suggests this idea isn't tied to one specific type of model architecture.

Lu: The convergence results are also strong; they show that even as you go to infinite tasks T infinity, the expected loss goes down and the iterates settle on a stable average teacher w star.

Tom: That means if you set it up right, long-term learning actually captures a stable representation of the data distribution rather than just chasing noise.

Jane: And they also show that this L2 regularization actually helps reduce label noise when you give the system enough tasks to see > Lu That's a nice side benefit to consider when dealing with noisy real-world data.

Lu: So, essentially, this paper gives us a theoretical roadmap for choosing the right amount of L2 regularization based on how much data—or tasks—or tasks—or data.

Tom: It turns hyperparameter tuning into a predictable scaling problem based on T over ln T > Jane and that's pretty exciting stuff for production systems.

Jane: And this leads us to think about how we can actually use this rule of thumb dynamically, so the regularization strength adjusts as the system learns more tasks.

The paper's summary: Tom: So, beyond just finding that T over ln T scaling law, the authors suggest we actually make things more dynamic when tuning that regularization strength lambda star.

Jane: They propose an adaptive regularization strength tuning method where you set lambda dynamically based on how many tasks you expect to learn > Lu I mean, it's not just a fixed number anymore.

Lu: Right. It suggests a way to adjust the stability versus plasticity balance based on the task horizon T, which should cut down on manual hyperparameter hunting significantly > Tom Yeah, that’s useful for production deployment because we don't want to be guessing at every single iteration.

Meng: If you can dynamically tune it based on T, that makes sense from a resource allocation standpoint; we're not wasting compute tuning a parameter that should scale with the problem size > Jane And this connects back to the noise robustness they mentioned earlier.

Jane: They also touch on how L2 regularization helps mitigate label noise when you have multiple iid teachers, suggesting that using this specific penalty term actually reduces the inherent noise floor from those different tasks > Tom So it’s not just about managing model complexity; it’s about improving data quality through the training method itself.

Lu: And they show that if you push lambda to infinity, even with an infinite number of tasks, the system converges to that global teacher w star, which means we get a really stable representation of the underlying data distribution > Meng So there's a guaranteed convergence point for long-term learning.

Tom: That’s a big win for long-horizon systems because it stops the model from just drifting off into some weird state after seeing thousands of tasks > Jane And they also give some practical advice on task ordering, suggesting you should end with the most typical task to keep teacher variability low > Lu That’s a great heuristic for data curation if you want to maximize generalization performance.

Jane: So, it’s not just one piece of math; it’s a whole strategy for designing continual learning systems that anticipates how long they'll run and what kind of noise they might encounter > Tom It really shifts the design focus from "how do I tune this once" to "how do I build a system that scales automatically with tasks."

The paper's improvements: Tom: So we've covered how this paper on Optimal L2 Regularization in High-dimensional Continual Linear Regression gives us a scaling law for lambda star, which is about T over ln T > Jane It’s basically telling us exactly how much regularization to use based on the number of tasks.

Jane: Yeah, and it also suggests we can make that tuning adaptive, so the system adjusts its stability versus plasticity as it learns more things > Lu That means the design changes from a one-time hyperparameter choice to a dynamic process that adapts to the learning progress.

Meng: From an engineer's view, if we can set up a rule of thumb based on T, it makes deploying these continual learning models much more predictable at scale > Tom I agree, it’s moving us away from just hoping our regularization works well enough on some test set.

Lu: The convergence proof is pretty strong; even with infinite tasks T infinity, the iterates settle onto a stable average teacher w star, provided the trace of the matrix sigma is zero > Jane That’s a solid result for long-term stability in these continual learning setups.

Lalam: I'm confident in this direction; it confirms that L2 regularization is a reliable way to handle those many teachers and noise we see across different tasks > Tom It’s pretty sure this scaling law holds up well even when we look at different model architectures, like linear regression versus neural networks with ReLU activations.

Jane: And the fact that it works across those architectures is really reassuring for practical applications because it doesn't rely on a specific type of deep learning structure > Lu So, to summarize, this paper provides a solid mathematical foundation for choosing regularization strength based on the task horizon T.

Tom: It’s a powerful tool we can use immediately when designing new continual learning systems that need to learn many things > Meng I just hope the engineering cost of implementing this adaptive tuning doesn't become too high for our production pipeline > Jane That’s a fair concern, but it seems like the theoretical payoff is worth exploring.

Lu: The next step would be testing these scaling laws on more complex, real-world scenarios with multimodal data to see if this holds up when things get messier > Tom Yeah, that’s where the future research goes > Jane And then we can look at how these ideas connect to other papers we've seen today about domain shift and feature refinement.

Conclusion: Tom: So we've gone through the mechanics of "Optimal L2 Regularization in High-dimensional Continual Linear Regression," which is essentially about figuring out how to set that lambda parameter when you keep learning new tasks without forgetting the old ones.

Jane: That scaling law, T over ln T, it’s a really concrete piece of math that gives us a rule of thumb for design rather than just some abstract idea > Lu I mean, it moves us from guessing to having a predictable approach.

Lu: Exactly. The authors show the expected generalization loss breaks down into label noise and teacher variability, which explains exactly why we need this specific regularization term to fight those issues > Meng That decomposition is useful because it tells you which source of error your penalty should be targeting when you tune lambda.

Tom: And they prove that this L2 scheme helps reduce label noise under both single-teacher and multiple iid teacher settings, which is a nice practical result for deployment > Jane It suggests that this specific regularization actually makes the system more resilient to those random changes in task ground truth over time.

Jane: And they also showed that when you push lambda really high, even with infinite tasks T infinity, the system converges to a stable average teacher w star, which is a strong statement about long-term stability > Lu That means we get a guaranteed convergence point for long-term learning sequences.

Meng: So if we use this rule of thumb based on T over ln T, it makes deploying these continual learning models much more predictable at scale because we don't have to waste time on hyperparameter tuning > Tom I agree, it’s moving us away from just hoping our regularization works well enough on some test set.

Jane: It really does give us a solid mathematical foundation for choosing that strength based on the task horizon T > Lu The next step would be testing these scaling laws on more complex, real-world scenarios with multimodal data to see if this holds up when things get messier.

Tom: Yeah, that’s where the future research goes. We've seen how this paper addresses one specific kind of continual learning problem, but we have to keep pushing it into more complex settings like those multimodal inputs Meng I just hope the engineering cost of implementing this adaptive tuning doesn't become too high for our production pipeline.

Jane: That’s a fair concern, but the theoretical payoff seems worth exploring because it gives us a way to design these systems that anticipates how long they'll run and what kind of noise they might encounter > Lu And we can look at how these ideas connect to other papers we've seen today about domain shift and feature refinement.

Lalam: I’m confident in this direction; it confirms that L2 regularization is a reliable way to handle those many teachers and noise we see across different tasks because it’s been rigorously proven > Tom It’s pretty sure this scaling law holds up well even when we look at different model architectures, like linear regression versus neural networks with ReLU activations.

Tom: It’s a powerful tool we can use immediately when designing new continual learning systems that need to learn many things > Meng That’s all for today. We'll keep an eye on how they apply this in complex scenarios where we have even more layers or multimodal inputs, and then we'll talk about how those ideas connect to feature refinement work.

Gilad Karpel, Edward Moroshko, Ran Levinstein, Ron Meir, Daniel Soudry, Itay Evron

Technion

cs.LG

Submitted: 2026-01-20

Updated: 2026-10-20

Comments: Accepted to ALT 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 86/100

The gist: The gist The expected generalization loss in an overparameterized continual linear regression setting, when analyzed under high-dimensional random design with multiple teachers and label noise, can

Key concepts

Generalization Loss Decomposition: The total expected loss after learning many tasks is broken down into four parts: label noise, teacher variability (mismatch between teachers), an interaction term reflecting update history, and temporal correlations between successive updates. Understanding these components helps in designing better regularization.
Optimal Regularization Scaling Law: The analysis finds the best L2 regularization strength ($\lambda\star$) scales approximately as T/ln(T) when teachers are independent and have a non-zero mean. This means the optimal strength depends on how many tasks (T) you expect to learn, offering a practical rule for tuning.
L2 Regularization Scheme: The L2 regularization term ($\lambda d\|w - w_{t-1}\|^2$) penalizes large changes in the model weights between successive updates. This helps control the plasticity of the learner, preventing it from drastically changing its parameters for every new task, which improves generalization.
Teacher Variability: This term measures how much the current teacher differs from past teachers due to task heterogeneity. High variability indicates that tasks are very different from each other, suggesting that regularization needs to be tuned carefully to manage this inherent mismatch.

Terminology

Regularization — 5× in this paper · explained in 3 episodes
Generalization — 3× in this paper · explained in 13 episodes
Continual Learning — 2× in this paper · explained in 3 episodes

Summary

The gist The expected generalization loss in an overparameterized continual linear regression setting, when analyzed under high-dimensional random design with multiple teachers and label noise, can be expressed as a closed-form expression that reveals an optimal regularization strength scaling nearly linearly with the number of tasks.

Setting and Model

The study focuses on continual linear regression where a learner is exposed to a sequence of T tasks, updating the predictor by fitting only the current task without access to previous samples. The regularization scheme analyzed is L2 regularization, defined as wt ← argminw n∥Xtw − yt∥2 + λd∥w − wt−1∥2. This analysis accommodates a setting broader than prior work by allowing both label noise and multiple teachers.

Generalization Loss Decomposition

The expected generalization loss E[GT] after learning T tasks is decomposed into four constituent terms. These terms are interpreted as follows:

Label noise term: arising from label noise of the underlying model, scaling with label variance.
Teacher variability: capturing the intrinsic mismatch between the current teacher and past teachers due to task heterogeneity.
Interaction term: reflecting how the deviation of the final teacher from the average teacher interacts with the sequence of teacher updates, discounting older changes.
Temporal correlations: accounting for correlations between successive teacher updates, describing how past changes compound and align (or misalign) over time.

Optimal Regularization Scaling Laws

The paper derives a closed-form expression for the expected generalization loss that accommodates diverse teacher setups. A key result is the characterization of the optimal regularization strength λ⋆, which scales nearly linearly with the number of tasks T. Specifically, for i.i.d. teachers and non-zero mean teacher w⋆, the optimal fixed regularization strength satisfies λ⋆ = λ⋆(T) ≈ T / ln T. This scaling law is crucial as it provides a practical rule of thumb for designing continual learning systems.

Validation and Scaling Behavior

The theoretical findings are validated through experiments on linear regression and neural networks. The empirical results show that the optimal regularization strength scales roughly as T / ln T, consistent with Theorem 6 even when the i.i.d. feature assumption does not hold. Furthermore, the analysis demonstrates that proper calibration of λ⋆ can substantially mitigate label noise after seeing enough tasks.

Asymptotic Convergence and Optimality

The study proves that as the horizon T → ∞, the expected generalization loss decreases monotonically with strength λ, and the iterates converge in mean square to the “global” teacher w⋆. The optimal regularization strength is shown to be near linear in T, specifically λ⋆ ∈ O(T) and λ⋆ ∈ omega(Tρ) for all 0 < ρ < 1. This establishes a robust theoretical foundation for choosing regularization strength based on the task horizon T rather than extensive hyperparameter tuning.

Related Work and Connections

The work extends the literature by incorporating both multiple teachers and L2 regularization, providing an exact average-case generalization analysis in a statistical setting. The use of a single regularization parameter (L2) is highlighted as a convenient analytical proxy for studying regularization strength compared to methods requiring many parameters. The findings offer a practical recipe for the design of continual learning systems.

The optimal regularization strength is shown to be near linear in T. This result suggests that tuning λ on a short task horizon and then scaling it for longer horizons can effectively determine the optimal strength. The analysis confirms that the qualitative prediction of Theorem 6—namely, that λ⋆(T) ≈ T / ln T—is consistent across both linear regression and neural network experiments. This consistency across architectures reinforces the practical utility of the derived scaling law. The overall conclusion is that any optimal regularizer λ⋆ must satisfy λ⋆ ∈ O(T) and λ⋆ ∈ omega(Tρ) for all 0 < ρ < 1.

Improvements for AI systems

Adaptive Regularization Strength Tuning: Implement a dynamic regularization strength λ that scales nearly linearly with task horizon T, specifically following λ⋆ = T / ln T for iid teachers with non-zero mean teacher w⋆. This allows the system to automatically adjust stability versus plasticity based on how many tasks it expects to learn, alleviating the need for manual hyperparameter tuning.
Noise Robustness Enhancement: The L2 regularization scheme mitigates label noise under multiple iid teacher settings, suggesting that incorporating this specific regularization term reduces the noise floor induced by label noise. This improvement makes the system more resilient when underlying task ground truth evolves randomly over time.
Optimal Task Ordering Guidance: The theoretical analysis suggests that learning should end with the most typical task to minimize teacher variability, as the teacher of the final task should be closest to the mean of all teachers w¯⋆. This provides a heuristic for sequencing tasks in continual learning systems to maintain better generalization performance.
Convergence Speed Under Strong Regularization: When employing strong regularization (λ → ∞), the system converges in mean square to the global teacher w⋆, even under infinite task horizons, achieving zero generalization loss if Tr(Σ) = 0. This ensures that long-term learning effectively captures a stable average representation of the underlying data distribution.
Architecture Agnostic Tuning: The theoretical results demonstrate that the optimal scaling law for regularization strength, λ⋆ ≈ T / ln T, is consistent across different architectures, including linear regression models and neural networks with ReLU activations. This provides a unified recipe for designing continual learning systems regardless of whether they use linear or deep non-linear structures.

Abstract

We study generalization in an overparameterized continual linear regression setting, where a model is trained with L2 (isotropic) regularization across a sequence of tasks. We derive a closed-form expression for the expected generalization loss in the high-dimensional regime that holds for arbitrary linear teachers. We demonstrate that isotropic regularization mitigates label noise under both single-teacher and multiple i.i.d. teacher settings, whereas prior work accommodating multiple teachers either did not employ regularization or used memory-demanding methods. Furthermore, we prove that the optimal fixed regularization strength scales nearly linearly with the number of tasks T, specifically as T/ T. To our knowledge, this is the first such result in theoretical continual learning. Finally, we validate our theoretical findings through experiments on linear regression and neural networks, illustrating how this scaling law affects generalization and offering a practical recipe for the design of continual learning systems.