Sequential Pretraining Favors Large Models

arXiv:2610.09611 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Sequential Pretraining Favors Large Models".

Jane: The gist: Exposure Therapy improves foundation models’ performance on late data distributions as well as overall capability in models up to the billion-parameter scale,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper called "Sequential Pretraining Favors Large Models," and it seems like the authors are asking a fundamental question about how big models learn compared to smaller ones.

Jane: It’s about whether the performance boost we see in these massive foundation models comes from them learning better representations or if it's just because they are more resilient to some kind of training bias.

Lu: The core idea is defining something called primacy bias, which is essentially how much seeing early data messes up the model's ability to learn later data.

Meng: So, the paper points out that small models struggle with this, while larger ones seem to handle it better because they have more capacity.

Lalam: It suggests that some of the benefits we attribute to scale might actually be due to this robustness against training issues rather than just learning more good features.

Tom: Exactly, and the paper sets up a comparison between target-first training and OOD-first training to quantify this impairment across different model sizes.

Jane: So, what they're doing is trying to figure out how that persistent gap in performance changes depending on whether you see the early data first or an out of distribution one.

Lu: The key finding here is that overparameterized models are far more robust to this primacy bias while small models are not >

Meng: So, if a model is smaller, it might just run out of room for the early data, which means it can't fit the later stuff like code or math.

Lalam: It means that in sequential pretraining where you see different data types one after another, small models really struggle to learn what comes later.

The paper's summary: Tom: They summarize their findings by focusing on how the persistent target-performance gap, which we call primacy sensitivity, changes as the number of parameters N increases.

Jane: They show that this sensitivity is positive when you have exposure to an out of distribution distribution during early training, meaning it persistently impairs the model’s fit on later distributions >

Lu: The paper shows that this impairment happens for small models because they can't allocate their learning capacity efficiently toward those early data distributions >

Meng: And the authors connect this inefficiency directly to why late-stage data is so important when you are training sequentially, like learning code or math.

Lalam: Basically, the paper argues that sequential pretraining constrains small models more severely because they get stuck optimizing for what they saw first >

Tom: So the main summary is that small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when those later data emphasize capabilities like code or reasoning >

Jane: It really boils down to the idea that it's not just about learning more features; it's about having the capacity to avoid being biased by early exposure >

Lu: The paper also introduces Exposure Therapy as a potential way to fix this capacity allocation problem during sequential pretraining, which they explore in detail >

The paper's improvements: Tom: Now for the improvements, the authors suggest that we can introduce Exposure Therapy as a regularization technique to help allocate learning capacity more efficiently throughout the training process >

Jane: Exposure Therapy involves sparsely replacing training updates with samples of downstream data distributions that the model would normally encounter later on >

Lu: This approach is different from replay methods because it focuses on improving performance related to newly encountered distributions instead of preserving capabilities learned earlier >

Meng: And they show that this Exposure Therapy can improve learning new data distributions as well as overall performance in foundation models, and it does this with negligible extra computational cost >

Lalam: They found that the minimal amount of Exposure Therapy needed to reach ninety percent of the target-first model’s performance decreases as the models get more overparameterized >

Tom: That suggests that we can recover some of that robustness previously gained through having more parameters, and this is done algorithmically >

Jane: So what this means for us is that Exposure Therapy improves acquisition of later distributions without sacrificing the useful structure learned earlier >

Conclusion: Tom: To wrap up, the paper "Sequential Pretraining Favors Large Models" argues that primacy bias reliably impairs models over several types of data distributions in sequential pretraining up to the billion-parameter scale >

Jane: They conclude that to help allocate learning capacity more efficiently across all data distributions, Exposure Therapy is a regularization that introduces a subset of late data early on where models show heightened plasticity >

Lu: It provides a cheaper alternative to just scaling up models and demonstrates how some benefits of model scale can be recovered through this algorithmic approach >

Meng: From an engineering standpoint, it’s interesting because it shows that we can get performance gains without needing massive increases in the number of parameters >

Lalam: I'm pretty sure they are right that this helps allocate learning capacity more efficiently, allowing smaller models to learn what they need later on >

Tom: So the main message is that for sequential pretraining, Exposure Therapy is a way to get better results without just throwing more parameters at the problem >

Mohnish Harwani, Yujia Zheng

Purdue University · University of Illinois Urbana–Champaign

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/MohnishHarwani/sequential_pretraining

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist: Exposure Therapy improves foundation models’ performance on late data distributions as well as overall capability in models up to the billion-parameter scale, suggesting some benefits of

Key concepts

Primacy Bias: This bias describes how exposure to early data distributions can negatively affect later learning. Small models struggle because they use up their limited capacity too quickly on initial data, making it hard for them to learn specialized skills presented later in training.
Overparameterized Models: These are very large models with many parameters. They are more robust to biases introduced during training, meaning they can better handle data distributions encountered early on and learn new, specialized distributions more effectively than smaller models.
Exposure Therapy (ET): ET is a regularization method that introduces a small subset of later training data samples early in the pretraining process. This helps the model focus its learning capacity on newly encountered data, improving performance without sacrificing previously learned capabilities.

Summary

The gist: Exposure Therapy improves foundation models’ performance on late data distributions as well as overall capability in models up to the billion-parameter scale, suggesting some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features.

Primacy Bias and Model Scale

Primacy bias is defined as the extent to which exposure to early data distributions impairs later learning on page 2. Small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect on page 2. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly on page 2. Small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning on page 2.

Mechanisms of Robustness

The results suggest that some benefits of large models may arise from greater robustness to biases introduced in training, as opposed to learning more representative solutions on page 2. In controlled vision experiments, overparameterized models are more robust to OOD data seen in early training, and are able to learn new distributions significantly better on page 4. This difference is not explained by a difference in plasticity in large models on page 4. Representational overlap analysis shows that Overparameterized models can largely circumvent capacity that fit early training distributions, and use other capacity they have available on page 4.

Sequential Pretraining Constraints

In foundation model pretraining, sequential data order constraints amplify the effect of primacy bias on page 4. Models may be exposed to different subsets of the training distribution in distinct stages, where early training may emphasize broad natural-language data, and later stages increasingly incorporate more specialized domains such as code, mathematics, tool use, and agentic behavior on page 4. We find that sequential training constraint favors large and overparameterized foundation models in terms of learning in late-stage data on page 4. Specifically, when small models lack the capacity to fit all data they encounter during training, learning new distributions increasingly requires repurposing neurons specialized to earlier data on page 4.

Exposure Therapy Intervention

Exposure Therapy (ET) is introduced as a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining on page 2. ET sparsely replaces training updates with a sample of downstream data distributions that models would otherwise encounter later in training on page 2. Unlike replay methods, ET focuses on improving performance related to newly encountered distributions, as opposed to preserving previously acquired capabilities on page 2. We demonstrate that Exposure Therapy "improves learning new data distributions as well as overall performance in foundation models, allowing models to recover part of the robustness otherwise obtained through additional parameters, with negligible extra computational cost" on page 2.

Exposure Therapy Efficacy

The minimal ET dose required to reach 90% of the target-first model’s performance decreases as models become more overparameterized on page 5. We evaluate ET at the billion-parameter scale across code, mathematics, German, and Finnish OOD corpora and the FineWeb and TinyStories English target distributions on page 7. Exposure Therapy improves both later-distribution and aggregate performance in 1B parameter models on page 7. ET therefore improves acquisition of the later distribution without sacrificing the useful structure learned earlier on page 7.

Ablation and Applicability

Ablation experiments show that Exposure Therapy improves pretraining in settings where assumptions may not extend to all pretraining setups on page 8. In reverse order pretraining, ET consistently reduces the impairment to English learning on page 8. Exposure Therapy also improves aggregate performance across all distributions in these settings on page 8. This demonstrates that its benefits persist when broad English data is encountered before more specialized distributions on page 8.

Conclusion

In sequential pretraining up to the billion-parameter scale, we find that primacy bias reliably impairs models over several types of data distributions on page 4. To help allocate learning capacity more efficiently across all data distributions, we introduce Exposure Therapy, a regularization that introduces a subset of late data early in pretraining where models may exhibit heightened plasticity on page 7. While adding negligible extra computational cost, this provides a cheaper alternative to scaling models, and demonstrates how some benefits of model scale can be recovered algorithmically on page 7.

Reproducibility Statement

We provide the code to toy vision experiments along with their mechanistic analysis, foundation model experiments, and Exposure Therapy, and additionally outlining hyperparameters and most experimental details for pretraining experiments within the Appendix on page 8. All datasets used are publicly available, and we include scripts to retrieve them on page 8.

References

Devansh Arpit, Stanisław Jastrz˛ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon LacosteJulien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, 2017. URL https://proceedings.mlr.press/v70/arpit17a.html on page 9.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, 2009. doi: 10.1145/1553374.1553380 on page 9.

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019 on page 9.

Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip H. S. Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021 on page 9.

Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical Japanese literature. arXiv preprint arXiv:1812.01718, 2018 on page 9.

Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632, 2024 on page 9.

Jeffrey L. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1), 1993 on page 9.

Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro.

Improvements for AI systems

Bold model robustness against early data bias: Improved foundation models will be more robust to adverse training effects like primacy bias because overparameterized models can circumvent capacity that fit early training distributions. This means smaller, capacity-constrained models will no longer struggle to learn distributions encountered later, such as code or mathematics.
Bold efficient learning capacity allocation: Implement Exposure Therapy (ET), a regularization technique where Exposure Therapy encourages small models toward parameters that remain compatible with later learning, thereby promoting more efficient allocation of learning capacity during sequential pretraining. This allows smaller foundation models to recover performance advantages previously only seen in larger models, with negligible extra computational cost.
Bold improved sequential pretraining for specialized tasks: When training on heterogeneous data sequentially, ET will ensure that the model learns late-stage distributions effectively because it addresses the issue where small foundation models can struggle to learn distributions encountered late in training. This is particularly beneficial when later data emphasizes desirable capabilities such as code, mathematics, and reasoning.