Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
summary
The gist
The gist The Split-on-Share (SoS) mechanism combined with adaptive elastic anchoring resolves the plasticity-stability conflict in continual learning by decomposing models into unique and shared
In short
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning introduces SETA, a framework that solves the plasticity-stability dilemma in LLMs during continual learning. It decomposes model knowledge into unique and shared experts using a Split-on-Share mechanism. This allows the model to learn new tasks without forgetting old ones by isolating task-specific patterns while protecting common features.
Key concepts
- Plasticity-Stability Dilemma
- This is the core conflict in continual learning: plasticity (the ability to learn new things) often causes instability (forgetting old knowledge). Standard methods treat all model parameters equally, which fails to manage this trade-off effectively.
- Split-on-Share (SoS) Mechanism
- This mechanism partitions model parameters into unique experts for new knowledge and shared experts for common features. It uses gradient topology overlap to decide which parameters become frozen unique memories and which are allowed to be shared, controlling how knowledge evolves.
- Task-Agnostic Inference
- SETA enables the model to perform any new task without needing explicit task identifiers during inference. It uses a unified router that selects the most relevant experts based on the input data, allowing for flexible and label-free adaptation.
Terminology used across episodes
This episode discusses
- Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning · Paper Radio
- On Tiny Episodic Memories in Continual Learning
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Towards a Unified View of Parameter-Efficient Transfer Learning
- LoRA: Low-Rank Adaptation of Large Language Models
- MeetingBank: A Benchmark Dataset for Meeting Summarization
- Domain Incremental Lifelong Learning in an Open World
- MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
- AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
- NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
- Resource Allocation for Dataflow Applications in FANETs using Anypath Routing
- GPT-4 Technical Report
- QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
- Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning
- Progressive Neural Networks
- Gradient Projection Memory for Continual Learning
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
- A Comprehensive Survey of Continual Learning: Theory, Method and Application
- TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
The paper
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning · Read on arXiv
Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari
Iowa State University · Argonne National Laboratory
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. SETA shares an overlapping parameter block only when the new task's update passes a subspace-agreement test and a sign test against its prior expert; shared knowledge is protected by adaptive elastic anchoring, and independently gated routers retrieve the relevant experts at inference without task identifiers. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, improving overall performance over the strongest baseline by 1.1 to 1.7 points on TRACE and by up to 5.5 points on the O-LoRA benchmark, with backward transfer between-2.2 and-1.3 on TRACE, against 4.6 to 6.3 points of forgetting for MoE-CL and 8 to 15 points for the other LoRA baselines.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning".
Jane: Here's the gist of the paper. The gist The Split-on-Share (SoS) mechanism combined with adaptive elastic anchoring resolves the plasticity-stability conflict in continual learning by decomposing models into unique and shared experts, thereby enabling task-agnostic inference.
Tom: Problem Formulation and Motivation Continual learning aims to enable models to acquire new capabilities from a sequence of tasks without forgetting previously learned knowledge (Page 2).
Jane: The core failure of continual learning arises from the Shared Subspace Dilemma, where parameters often become important for multiple tasks, leading to an intersection I = Pt ∩ P1:t−1 (Page 2).
Tom: Standard methods like uniform regularization are too rigid because they treat parameters equally rather than distinguishing shared features requiring controlled plasticity from task-specific regions needing stability (Page 2).
Paper summary: Tom: So, thinking about the title, "Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning," what this paper really boils down to is using a structured decomposition of parameters to keep learning new skills without erasing old ones.
Jane: It’s about taking that plasticity stability problem and solving it by creating two sets of experts—unique ones for new, task-specific knowledge, and shared ones for common features that need to stay stable.
Lu: The core idea is this subspace-based expert decomposition which frames sparse continual learning within a Mixture-of-Experts paradigm, which lets the model operate in isolated subspaces instead of all parameters competing.
Meng: What this implies for us at the startup level is that if we can find a good way to implement those unique and shared partitions, we can train models that adapt sequentially without needing constant retraining from scratch or massive parameter updates.
Lalam: And it means for users, they get a model that just keeps getting smarter over time without you having to manually manage the knowledge retention parts of the system.
Tom: The paper shows this framework is effective, achieving competitive results and improving performance on benchmarks like TRACE by up to one point seven points and showing much better backward transfer than other methods.
Jane: Overall, SETA gives us a way to build LLMs that can learn new things incrementally while retaining their core capabilities, all handled through this clever split-on-share mechanism.
Conclusion: Tom: So, we're wrapping up this deep dive into Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning by SETA, and I want to talk about what that title actually means in plain English.
Jane: It means they found a way to split the model’s knowledge so that when you teach it something new, it doesn't forget the old stuff.
Tom: Exactly. They used this mixture of experts idea, but instead of just having all parameters fight for every task, they made unique experts for new things and shared ones for common features.
Lu: The mechanism is called Split-on-Share, where they look at how much the current learning direction overlaps with what has been learned before.
Meng: From a deployment standpoint, that separation sounds like it could make updates more controlled at scale because you aren't touching everything simultaneously.
Jane: But the big payoff here is task-agnostic inference, which means you don't need to know what task you’re working on when the model makes a decision.
Tom: Right. It uses these separate routers to pick which expert—the unique one or one of the shared ones—to use for every input, without needing a label.
Lalam: I can confirm that by separating those knowledge components into distinct experts, they prevent parameter overwriting which is what usually causes forgetting.
Jane: And the numbers they show are pretty telling too. They're seeing performance gains on TRACE and O-LoRA benchmarks, like up to one point seven points on TRACE.
Lu: That’s solid evidence that this structured approach to parameter decomposition actually works better than just standard uniform regularization methods.
Tom: It shows that we don't have to treat every single weight the same when we're building these continual learning systems, which is a key mechanical win here.
Jane: So, the implication for us is that instead of constantly fine-tuning or risking catastrophic forgetting, we might be able to build models that just grow incrementally.
Lu: It opens up possibilities for cumulative learning across many different domains without needing a completely new architecture every time we add a task.
Tom: Exactly. So, if this approach holds up when we scale it to bigger models and longer sequences, it could change how we think about long-term model adaptation in production.
Jane: Next time on the show, we're going to look at how those unique and shared experts actually evolve over time.
More episodes
- 2505.16051-Generative Flows for Potential Outcomes and Counterfactuals
- 2505.19458-Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
- 2506.02685-Symmetry-Aware GFlowNets
- 2506.03979-Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approach
- 2506.19384-Deep Electromagnetic Structure Design Under Limited Evaluation Budgets
- 2508.05995-Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization
- 2508.15215-Sleep Stage Classification across Heterogeneous EEG/EOG Channels with Multi-Channel Differential Transformer
- 2509.00990-Hybrid Topic-Semantic Labeling and Graph Embeddings for Unsupervised Legal Document Clustering
- 2509.12057-Optimal hypersurface decision trees
- 2509.14229-Spacing Test for Fused Lasso