Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

The gist The authors analyze single-pass Shuffle SGD under strongly convex and general convex objectives to derive explicit upper bounds on the expected final forgetting in online continual learning, revealing how data amount, task similarity, and task ordering govern this phenomenon across different objective classes.

Theoretical Framework

The analysis formalizes forgetting by defining the model error on task i as Ki(w):= w − w∗i 2 (Page 3). The authors establish recursion relations for the expected squared distance between the current model wt and a previous optimum w∗i, which involves terms related to task gaps and propagation factors q or ρ (Page 4). For strongly convex objectives, the propagation factor is defined as q = 4(1 − γµ)n and the variance constant as V = 4γσ2S/µ (Page 12). In the general convex case, they introduce tunable Young coefficients η, θ > 0 to define a one-step growth factor ρ = (1 + η)(1 + θ) > 1 (Page 3).

Governing Factors

The paper explicitly examines three factors that govern these bounds:

1. Data amount: For strongly convex objectives, the bound depends on n through the propagation factor q(n), which decreases geometrically in n, and the variance constant V(n), which grows linearly in n for large n (Page 5). For general convex objectives, data amount enters only through an accumulated variance term that grows with n within the stepsize range of the analysis (Page 5).

2. Task similarity: This is quantified by the optimal-model gap ∆i,j:= w∗i −w∗j 2 (Page 5). The bounds depend on this gap, and a larger gap means lower similarity, but this effect is mediated by ordering coefficients which couple similarity with task ordering (Page 5).

3. Task ordering: The order in which tasks are learned affects the bounds through the order-dependent coefficients ci,j and gi,j (Page 5). For instance, when q < 1 in the strongly convex case, spacing similar tasks apart is favored to minimize the bound (Page 6).

Empirical Validation

Experiments with MLPs on Split-CIFAR-100 and PMNIST show that the trends suggested by the theory appear in practice (Page 2). The authors report results for forgetting, average accuracy (ACC), and backward transfer (BWT) across varying data amount, task similarity, and task ordering (Page 6). Specifically, less data per task tends to yield higher accuracy and better backward transfer in these experiments (Page 8). Furthermore, the results confirm that a less similar target task lowers the final accuracy and increases forgetting because the gap between two tasks always enters the bounds with a nonnegative weight (Page 7).

Conclusion

The work provides explicit upper bounds on expected forgetting for OCL under both convex objective classes, demonstrating how data amount, task similarity, and task ordering interact to shape these bounds (Page 7). The empirical evidence supports the qualitative role of these factors in understanding online continual learning behavior (Page 7). The framework suggests that the identified factors offer useful guidance for understanding OCL behavior (Page 7). Future work includes extending the analysis to structured non-convex settings and turning ordering coefficients into practical scheduling algorithms (Page 13).

--- Page 1 ---

Renjie Cao Yifei Cheng Li Shen† Boston College caoren@bc.edu Sun Yat-sen University chengyf28@mail.sysu.edu.cn Sun Yat-sen University mathshenli@gmail.com Abstract Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge; existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting; Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes; Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting. 1 INTRODUCTION In online continual learning (OCL), a learner is trained on a non-stationary stream of tasks and observes each sample only once (Aljundi et al., 2019); The central challenge is to balance plasticity, the ability to learn new tasks, with stability, the ability to retain previously learned ones; When the model adapts too aggressively to incoming data, its performance on earlier tasks can degrade severely, a phenomenon known as catastrophic forgetting (McCloskey and Cohen, 1989); Forgetting is particularly pronounced in the online setting, because † Corresponding author. the single-pass constraint prevents the learner from revisiting past samples; Many methods have been proposed to address this stability–plasticity dilemma; Rehearsal-based methods maintain a small buffer of past samples (Harun et al., 2024; Wu et al., 2024) or replay generated features of past tasks (Li et al., 2025a); Other approaches protect useful parameters through utility-based gradient perturbation (Elsayed and Mahmood, 2024) or use self-distillation (Nagata et al., 2025) to preserve past representations; More recently, parameter-efficient fine-tuning with online low-rank adaptation (LoRA) (Wei et al., 2025) and contrastive replay with dual margins (Lyu et al., 2026) have been used to reduce interference between tasks. In parallel, theoretical work has begun to characterize forgetting dynamics; for instance, Shan et al. (2026) develop a statistical-mechanics theory that identifies phase transitions of continual learning in deep networks; Nevertheless, analytical frameworks that express the expected forgetting explicitly in terms of interpretable factors remain scarce; Understanding which factors govern forgetting is therefore essential, yet such analyses are still rare. The work most closely related to ours is Lin et al. (2023), which derives exact expressions for the expected forgetting of overparameterized linear models and highlights the roles of overparameterization, task similarity, and task ordering; We pursue the same goal of making forgetting analytically interpretable, but study single-pass Shuffle SGD under general convex and strongly convex objectives; This distinction matters because the mechanisms that govern forgetting change with the contraction or growth properties of the objective: in our analysis, data amount, task similarity, and task ordering remain the central factors, but their effects are mediated by the objective class through the coefficients of the bounds. Our theoretical results are as follows. (1) Data amount: arXiv:2610.14787v1 [cs.LG] 10 Oct 2026

--- Page 2 ---

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning for strongly convex objectives, increasing the per-task data amount n decreases the propagation factor of the bound, so the task-gap part of the bound decays exponentially in n, while its variance part grows linearly in n through the shuffling variance; for general convex objectives, n enters only through an accumulated variance term, which grows with n within the stepsize range of the analysis. (2) Task similarity: the gap between task optima enters the bounds through nonnegative, order-dependent coefficients, so a larger gap never decreases the bound, while its weight depends on where the two tasks appear in the sequence. (3) Task ordering: The coefficients favor spacing similar tasks apart when the propagation factor of the strongly convex bound is below one, and placing them close to each other otherwise, including the general convex case. Experiments with MLPs on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting. 2 RELATED WORK Empirical methods for OCL. Most OCL methods rely on rehearsal, regularization, or the adaptation of pre-trained models; Rehearsal-based methods select which samples to store or replay (Aljundi et al., 2019; Harun et al., 2024), stabilize representations and optimization under replay (Caccia et al., 2022; Yoo et al., 2024), model the relations among previous tasks (Wu et al., 2024), or select the knowledge relevant to the current task (Han and Liu, 2025); Other approaches protect useful parameters (Elsayed and Mahmood, 2024), distill past knowledge (Nagata et al., 2025), average the model with earlier checkpoints (Kleiman et al., 2025), or adapt pre-trained models through low-rank adaptation and prompts (Wei et al., 2025; Wang et al., 2025). Empirical studies further examine how model size (Lee, 2024), limited computation (Cai et al., 2023), synthetic data contamination (Wang et al., 2019a), and model throughput (Wang et al., 2019b) affect OCL, and extend it to embodied agents (Kim et al., 2024). In contrast to these algorithmic and empirical studies, we ask how properties of the task stream itself govern forgetting. Theoretical analysis of forgetting.

1. Rough Data Scaling for Task Retention in Single-Pass SGD: Provide explicit guidance on data amount impact by analyzing how more data per task does not automatically lead to better retention. This allows system designers to understand when increasing sample size will exacerbate forgetting, which is critical for designing efficient single-pass learning protocols.

2. Optimal Task Ordering Strategy via Gap Minimization: Implement a dynamic scheduling mechanism that prioritizes placing similar tasks close together when the propagation factor "q > 1, as this is favored by the analysis for minimizing the bound. Conversely, if q < 1," move similar tasks farther apart to minimize forgetting, directly translating theoretical results into practical task sequencing rules.

3. Task Similarity Metric Refinement: Use optimal-model distance as a quantifiable surrogate for task relatedness, ensuring that feature representation learning prioritizes maintaining small gaps between the optima of sequentially learned tasks. This guides the system to learn representations where a smaller gap indicates more similar tasks with closer optima.

Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.