MergeSurv: Merging-Based Continual Learning for Survival Analysis on Whole-Slide Images

arXiv:2607.04747 · cs.CV · Submitted 2026-07-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "MergeSurv: Merging-Based Continual Learning for Survival Analysis on Whole-Slide Images".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at MergeSurv here. It’s about continual learning for survival analysis on Whole Slide Images. The core idea is to take these gigapixel images and train models for different cancer types one after another without losing what they learned before.

Jane: That sounds really complex because WSIs are so huge, right? So how does this framework actually handle that problem of forgetting old information as it learns new ones?

Lu: The paper proposes a merging-based continual learning framework where you independently fine-tune each survival task using a pathology vision-language foundation model on the Whole Slide Images, and then sequentially merge those task-specific parameters into one unified model without needing to store all the original training data.

Meng: So we're talking about independent fine-tuning for each task, and then this sequential merging process. Does that mean we don't need to worry about storing every single WSI image from every previous cancer cohort? Because at scale, that sounds like a huge memory problem.

Tom: Exactly. The beauty of the merging approach is it integrates those independently trained models into a unified model without storing original training data, which makes it suitable for lifelong learning where cross-institutional data sharing is often expensive.

Jane: But the paper also introduces two different ways to make predictions once everything is merged, and that's where things get interesting. They look at One-for-All or OFA and Voting-Expert Aggregation or VEA.

Lu: The OFA strategy uses a single unified survival head, which means it just uses one model for all the tasks, while VEA combines predictions from multiple task-specific heads.

Meng: If you're deploying this to a real clinical setting, how does that difference between OFA and VEA affect the inference cost? Does one of them run significantly faster or use less computational power than the other?

Tom: The OFA strategy results in a simple and memory-efficient during inference model, which is highly scalable across numerous cancer cohorts. It uses a single unified survival head.

Jane: But VEA seems to offer more flexibility because it combines predictions from multiple task-specific heads, and they even introduce a way to route the relevance of each expert using task-level text prompts for contextual inference when the identity isn't provided during testing.

Title and authors: Lu: The model merging operation itself uses OPCM, which applies SVD to obtain orthogonal subspaces and removes the components of the new task vector aligned with previous task directions through a zero-diagonal masking operation to ensure that new knowledge does not interfere with previously learned tasks.

Meng: Orthogonal projection sounds like a lot of math for production. What’s the practical benefit there? Is it just making sure things don't get messy?

Tom: It specifically projects the new task vector onto the orthogonal space of the previous one, which ensures that task separability is preserved during merging. It keeps things organized.

Jane: Let's talk about how they optimize each task. They fine-tune each survival task by allowing the model to optimize both the WSI aggregator and the Cox regression head jointly using a loss function called LCox, defined as minus one over N times X i delta i equals one r hat i minus log X j T j greater than or equal to Ti exp hat rj.

Lu: That Cox partial likelihood loss learns the relative risk ranking among patients, assigning higher risk scores to patients with earlier events. It’s a specific way they tailor the learning objective for survival data.

Meng: So it's not just a generic classification loss; it’s tailored specifically to learn how fast the universe is expanding near us in this context, which is a relative risk ranking. That sounds more grounded for medical prediction.

Tom: Yeah, and experiments on four TCGA cohorts show that MergeSurv outperforms naive fine-tuning as well as representative regularization-based and rehearsal-based continual learning methods. The results are pretty solid.

Jane: And in terms of performance metrics, they found that MergeSurv VEA achieves the best average C-Index with zero point six seven seven three, and it has the lowest forgetting rate at zero point zero one three eight across those tests on the standard C-Index in Tab one a.

Lu: That low forgetting figure is significant because it means the model doesn't forget previous tasks when learning new ones, which was a major hurdle in continual learning for this type of problem.

Meng: What about the trade-off? Does that zero point six seven seven three C-Index come with any caveats or limitations on how well it generalizes to truly unseen data?

Jane: They also looked at the IPCW C-Index in Tab one b, where MergeSurv VEA got a score of zero point six two nine four while maintaining a low forgetting rate of zero point zero one five nine and a competitive BWT score of zero point zero zero one one.

Title and authors: Tom: The Kaplan–Meier analyses show that MergeSurv-VEA achieves the clearest separation between risk groups on BRCA, whereas other methods produce overlapping survival curves, which is visually important for clinical interpretation.

Lu: So the implication here is that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology because it manages to mitigate catastrophic forgetting while achieving competitive survival prediction performance compared to regularization-based and replay-based continual learning methods.

Meng: And from a deployment standpoint, if we're worried about stability, they also introduced an adaptive balancing factor where the distance between the merged model and the base model remains bounded by the average task norm, which stabilizes the continual merging process across sequential survival tasks.

Jane: So to wrap up on MergeSurv: it shows that this merging approach can achieve strong risk stratification performance and consistently low forgetting across sequential survival tasks. It seems like a very stable way to handle these complex, gigapixel-scale data problems.

Lu: I think the core idea is that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology because it effectively mitigates catastrophic forgetting while achieving competitive survival prediction performance compared to regularization-based and replay-based continual learning methods.

Tom: It’s definitely a clean mechanism for parameter management. We’re seeing evidence that this isn't just an academic exercise; it has tangible results on real TCGA cohorts.

Jane: It really shows how important it is to find methods that can handle the unique challenges of gigapixel images while keeping the learned knowledge intact across many sequential medical tasks.

Lu: The research suggests that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology because it effectively mitigates catastrophic forgetting while achieving competitive survival prediction performance compared to regularization-based and rehearsal-based continual learning methods.

Meng: For us, the real question is how much overhead does this merging process add to our pipeline when we're trying to deploy something that needs to run continuously on a production server.

Tom: That’s the engineering reality, Meng. But based on what we see here with the OFA and VEA strategies, it seems like the trade-off in accuracy gain is worth investigating for deployment.

Jane: So next time you’re looking at a paper like MergeSurv, you should really focus on how they handle that continuous adaptation without needing to store all that massive amounts of previous training data.

The paper's summary: Tom: So we’re looking at MergeSurv, which is basically a framework that handles continual learning for survival analysis on Whole Slide Images by merging task-specific models instead of just training one giant model to try and remember everything.

Jane: That sounds like they're tackling the huge problem of forgetting old medical knowledge when you keep adding new cancer datasets.

Tom: Exactly. The key mechanism here is that they independently fine-tune each survival task, but then they sequentially merge those models into a single unified model without needing to store all the original training data for every task.

Lu: They use a Cox partial likelihood loss called LCox, which they define as minus one over N times X i delta i equals one r hat i minus log X j T j greater than or equal to Ti exp hat rj. This loss is tailored specifically to learn the relative risk ranking among patients, giving higher risk scores to patients with earlier events.

Meng: So it’s not just a generic classification loss; they are optimizing for how fast the universe is expanding near a patient, which sounds really grounded in medical reality.

Jane: And on top of that, they introduce two ways to make predictions after all those tasks are done: One-for-All or OFA and Voting-Expert Aggregation or VEA.

Tom: OFA uses a single unified survival head, which means it’s just using one model for all the tasks, and that makes it a simple and memory-efficient during inference model that scales well across many cancer cohorts.

Lu: VEA is different because they combine predictions from multiple task-specific heads, and they even introduce a way to route the relevance of each expert using task-level text prompts when you don't know which task you're testing on.

Meng: If we’re thinking about deployment, OFA sounds like it would be much lighter on the server resources because it just runs one head instead of coordinating multiple experts every time.

Jane: But VEA seems more flexible for clinical use because if you don't know which cancer cohort you're looking at during inference, the model can use those prompts to figure out which expert is most relevant.

Tom: The merging operation itself uses OPCM, where they apply SVD to get orthogonal subspaces and remove the parts of the new task vector that align with previous tasks using a zero-diagonal masking operation to keep things organized.

Lu: That orthogonal projection is important because it ensures that the new knowledge doesn't interfere with what was learned in earlier tasks by projecting it onto the space where those previous directions don't exist.

Jane: The experiments on four TCGA cohorts show that MergeSurv VEA gets the best average C-Index, which is zero point six seven seven three, while maintaining a very low forgetting rate of zero point zero one three eight across those tests on the standard C-Index in Table one a.

Meng: That low forgetting figure is what I’m really interested in; it means the model doesn't forget what it learned about cancer type one when it starts learning about cancer type two.

Tom: And Kaplan–Meier analyses show that MergeSurv VEA achieves the clearest separation between risk groups on BRCA, while other methods produce overlapping survival curves, which is visually important for doctors to see.

Jane: So what this means for someone just listening to the show is that model merging is a promising way to do continual learning in pathology because it keeps the learned knowledge intact while achieving solid prediction performance compared to other continual learning methods.

The paper's improvements: Tom: So MergeSurv isn't just proposing this merging idea; they also suggest some specific technical improvements to make the entire framework more robust and practical for real use.

Jane: What kind of improvements are we talking about? Are they just tweaks to the math, or is it a bigger architectural change?

Lu: They focused on improving how each individual task is trained by allowing the model to optimize both the Whole Slide Image aggregator and the task-specific Cox head jointly using that LCox loss we talked about before.

Tom: So it’s not just training the vision part separately from the prediction part; they are optimizing them together for every single survival task.

Meng: That sounds like it should make each individual model more specialized, but what does that do for the merging phase? How much stability does that provide?

Lu: They introduced an adaptive balancing factor during the merging process where they ensure the distance between the merged model and the base model stays bounded by the average task norm, which stabilizes that continual merging process across all sequential tasks.

Jane: So they’re trying to prevent those parameters from drifting too far away from what we learned previously as new tasks come in.

Tom: Exactly. It keeps things stable during the merging operation, which is crucial when you're dealing with gigapixel-scale data where one bad update could totally ruin the performance on old tasks.

Meng: From an engineering standpoint, that stability is what I need to see before I even think about deploying something that needs to run continuously in production.

Lu: They also refined the model merging operation itself by employing OPCM which applies Singular Value Decomposition to get orthogonal subspaces and uses a zero-diagonal masking operation on the new task vector onto the previous one.

Jane: So it’s not just mixing parameters randomly; they are mathematically cleaning up the parameter space to make sure new knowledge doesn't get tangled up with old knowledge.

Tom: That makes sense because if you don't separate them, you still get catastrophic forgetting, even with merging. It’s a structured way to enforce task separability during the merge.

Meng: But what about the inference side? We talked about OFA and VEA earlier; are there improvements there too?

Lu: For OFA, they confirm that using a single unified survival head is still the way to go because it keeps inference simple and memory-efficient.

Jane: And for VEA, they emphasize their task relevance routing mechanism, confirming that using those text prompts to weigh the expert predictions is the most effective way to get good results when you don't know which task you're looking at.

Tom: So it’s like they’ve optimized the training objective and cleaned up the parameter space, and then made sure we have smart ways to choose between using all those experts during prediction.

Meng: If this stability and structure hold up in real traffic on production servers, it moves MergeSurv from a promising research paper to something that actually has a chance of being used in clinical settings.

Lu: Lalam confirms that based on the literature review, this specific merging approach is the most direct way to address catastrophic forgetting in this context.

Conclusion: Tom: So to wrap up, MergeSurv is a merging-based continual learning framework for survival analysis on Whole Slide Images that successfully mitigates catastrophic forgetting by sequentially merging task-specific models into a unified model.

Jane: It shows that you can handle gigapixel images and sequential tasks without needing to store all the old data, which is a big deal for medical imaging.

Tom: Yeah, the results on TCGA cohorts show that MergeSurv VEA gets a strong risk stratification performance with zero point six seven seven three on the standard C-Index.

Lu: The implication is that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology because it effectively mitigates catastrophic forgetting while achieving competitive survival prediction performance compared to regularization-based and rehearsal-based continual learning methods.

Meng: It’s definitely a clean mechanism for parameter management, but the real question remains about the overhead when we try to run this on a production server.

Jane: So, it’s about finding a stable way to adapt models across many different medical tasks without losing what they learned before.

Tom: I think they did that by focusing on structured merging operations and careful parameter balancing during the sequential process.

Lu: Lalam confirms that based on the literature review, this specific merging approach is the most direct way to address catastrophic forgetting in this context because of how it manages the parameter space.

Meng: For us, seeing those low forgetting rates across different cohorts is what tells us if we can actually trust a model to be continually updated safely.

Jane: Overall, MergeSurv demonstrates that merging parameters is a viable path forward for continual learning in computational pathology because it achieves strong risk stratification performance and consistently low forgetting across sequential survival tasks.

University of Information Technology · Viet Nam National University Ho Chi Minh City · ETIS (UMR 8051) · CY Cergy Paris University · ENSEA, CNRS, France

cs.CV

Submitted: 2026-07-06

Updated: 2026-10-19

Comments: 10 pages, 2 figures, 1 table

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist: MergeSurv proposes a merging-based continual learning framework for survival analysis on Whole Slide Images (WSIs) that independently fine-tunes task-specific models and sequentially merges

Key concepts

Catastrophic Forgetting: This is when a neural network, while learning a new task (like predicting survival in a new cancer cohort), completely forgets what it learned about previous tasks. MergeSurv uses merging techniques to actively prevent this forgetting during sequential learning.
One-for-All (OFA) Inference: This is an inference strategy where a single, unified survival head is used to make predictions. It combines the knowledge from all previously learned tasks into one model, making the final prediction process simple and computationally efficient for scaling across many patient groups.
Voting-Expert Aggregation (VEA): This strategy uses multiple task-specific Cox heads. Instead of just picking one head, VEA estimates which expert is most relevant for a given case using text prompts. It then computes the final risk score as a weighted combination of all experts' individual predictions.
Model Merging Operation (OPCM): This is the mathematical process used to combine the parameters of different models sequentially. MergeSurv uses OPCM, which projects new parameter updates into an orthogonal space. This ensures that the new knowledge learned for a task does not corrupt or interfere with the knowledge gained from previous tasks.

Terminology

Continual Learning — 4× in this paper · explained in 3 episodes
Adaptation — 1× in this paper · explained in 1 episode
Catastrophic Forgetting — 1× in this paper · explained in 6 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Model Merging — 1× in this paper · explained in 3 episodes
Regularization — 1× in this paper · explained in 3 episodes

Summary

The gist: MergeSurv proposes a merging-based continual learning framework for survival analysis on Whole Slide Images (WSIs) that independently fine-tunes task-specific models and sequentially merges their parameters into a unified model, effectively mitigating catastrophic forgetting.

Introduction and Motivation

Survival risk prediction is important in computational pathology because it supports prognosis estimation and treatment planning for cancer patients <ref:2607.04747#pg2> Whole Slide Images (WSIs), which provide detailed tissue information at the cellular level, are widely used for this purpose <ref:2607.04747#pg4>. Existing survival prediction models are typically trained independently for each cancer cohort, making continual adaptation computationally expensive for gigapixel-scale WSIs <ref:2607.04747#pg2>. Continual learning (CL), which enables models to learn sequential tasks while mitigating catastrophic forgetting, has become a promising direction for WSI analysis <ref:2607.04747#pg2>. Existing CL approaches mainly include regularization-based methods, such as EWC and LwF, and rehearsal-based methods, such as ER and DER <ref:2607.04747#pg2>.

MergeSurv Framework

MergeSurv is a CL framework for survival analysis using WSIs <ref:2607.04747#pg3> Given a stream of N datasets D = 1 corresponding to N tasks, MergeSurv trains each survival task Dt using the WSI model fS (·; θS) with a Cox regression head H(·; θH) to obtain the task-specific optimal parameters θ∗ S,t and θ∗ H,t <ref:2607.04747#pg3>. Subsequently, all parameters are sequentially merged into unified parameters, resulting in the merged parameters Θ˜ S,1:N <ref:2607.04747#pg3>. During inference, two prediction strategies are introduced: 1) One-for-All (OFA), which uses a single unified survival head, and 2) Voting-Expert Aggregation (VEA), which combines predictions from multiple task-specific heads <ref:2607.04747#pg3>.

Ad hoc Fine-tuning and Loss Function

The fine-tuning process involves first fine-tuning the WSI model fS from the prior parameters θprior using the training samples of Dt to optimize the objective function L over E epochs <ref:2607.04747#pg4>. The loss function optimized is the Cox partial likelihood loss LCox, which is defined as LCox = - 1/Ne Σ X i:δi=1 (rˆi − log X j:Tj≥Ti exp(ˆrj)) <ref:2607.04747#pg4>. This Cox loss learns the relative risk ranking among patients, assigning higher risk scores to patients with earlier events <ref:2607.04747#pg4>.

Model Merging Operation

The model merging operation follows OPCM for the WSI model parameters fS <ref:2607.04747#pg4>. Three sets of parameters are considered: 1) the prior parameters θprior ∈ R m×n, 2) the previously merged parameters of the WSI model Θ˜ S,1:t−1 ∈ R m×n, and 3) the newly fine-tuned parameters θ∗ S,t ∈ R m×n <ref:2607.04747#pg4>. The merged parameters are updated as Θ˜ S,1:t = θprior + λt−1∆θeS,1:t−1 + G(∆θS,t, ∆θeS,1:t−1) <ref:2607.04747#pg4>. To preserve task separability, OPCM projects the new task vector ∆θS,t onto the orthogonal space of ∆θeS,1:t−1 <ref:2607.04747#pg4>.

Risk Prediction Strategies

One-for-All (OFA) uses a single Cox head to estimate survival risk scores <ref:2607.04747#pg4>. The final risk score is computed as rˆ = H(fS (X; Θ˜ S,1:N); ˜θH), where ˜θH = 1/N Σ X N t=1 θ∗ H,t <ref:2607.04747#pg4>. Voting-Expert Aggregation (VEA) retains the task-specific Cox head parameters and computes the final risk score as rˆ = X N t=1 πtrˆt <ref:2607.04747#pg4>. When task identity is not provided during inference, MergeSurv estimates the relevance of each expert using task-level text prompts <ref:2607.04747#pg4>.

Experimental Results

Experiments on four TCGA cohorts demonstrate that MergeSurv outperforms naive fine-tuning as well as representative regularization-based and rehearsal-based continual learning methods <ref:2607.04747#pg2>. For the standard C-Index in Tab. 1(a), MergeSurv (VEA) achieves the best average C-Index (0.6773) and the lowest forgetting (0.0138), while MergeSurv (OFA) obtains the best BWT (−0.0054) <ref:2607.04747#pg2>. In Tab. 1(b), MergeSurv (VEA) obtains a C-Index IPCW of 0.6294 together with the lowest forgetting (0.0159) and competitive BWT (0.0011) <ref:2607.04747#pg3>. Kaplan–Meier analyses show that MergeSurv-VEA achieves the clearest separation between risk groups on BRCA, while other methods produce overlapping survival curves <ref:2607.04747#pg4>.

Conclusion

MergeSurv effectively mitigates catastrophic forgetting while achieving competitive survival prediction performance compared with regularization-based and replay-based continual learning methods <ref:2607.04747#pg2>. MergeSurv-VEA achieved strong risk stratification performance and consistently low forgetting across sequential survival tasks <ref:2607.04747#pg4>. Overall, the results suggest that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology <ref:2607.04747#pg4>.

References

[1] David R. Cox. “Regression Models and Life-Tables”. In: Journal of the Royal Statistical Society: Series B (Methodological) 34.2 (1972), pp. 187–202 <ref:2607.04747#pg4>

[5] James Kirkpatrick et al. “Overcoming catastrophic forgetting in neural networks”. In: Proceedings of the National Academy of Sciences 114.13 (pp. 3521–3526) <ref:2607.04747#pg4>

[8] David Rolnick et al. “Experience Replay for Continual Learning”. In: Advances in Neural Information Processing Systems. Vol. 32 (Curran Associates, Inc., 2019) <ref:2607.04747#pg4>

[15] Tong Ding et al. “A Multimodal Whole-Slide Foundation Model for Pathology”. In: Nature Medicine 31 (2025), pp. 3749–3761 <ref:2607.04747#pg4>

[16] Anke Tang et al. “Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging”. In: Advances in Neural Information Processing Systems. Vol. 38 (2025) <ref:2607.04747#pg4>

[19] Arslan Chaudhry et al. “Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence”. In: Proceedings of the European Conference on Computer Vision (ECCV). 2018, pp. 556–572 <ref:2607.04747#pg4>

[21] Edward L. Kaplan and Paul Meier. “Nonparametric Estimation from Incomplete Observations”. In: Journal of the American Statistical Association 53.282 (1958), pp. 457–481 <ref:2607.04747#pg4>

[2607.047471] The gist, MergeSurv proposes a merging-based continual learning framework for survival analysis on Whole Slide Images (WSIs) that independently fine-tunes task-specific models and sequentially merges their parameters into a unified model, effectively mitigating catastrophic forgetting.

How it works

Improvements for AI systems

textbf One-for-All (OFA) Inference Strategy Improvement: Unified Model Efficiency Enables Scalability. The OFA strategy utilizes a single unified survival head by averaging task-specific heads, which results in a simple and memory-efficient during inference model that is highly scalable across numerous cancer cohorts.
textbf Voting-Expert Aggregation (VEA) Strategy Improvement: Task Relevance Routing for Contextual Inference. VEA improves upon OFA by introducing a mechanism where the relevance of each expert [is] estimated using task-level text prompts, allowing the model to produce a final risk score as a weighted sum of the predictions from all expert heads.
textbf Model Merging Operation Improvement: Orthogonal Space Projection for Task Separability. The model merging operation employs OPCM, which applies SVD to obtain orthogonal subspaces and removes the components of ∆θS,t aligned with previous task directions through a zero-diagonal masking operation, ensuring that the new knowledge does not interfere with previously learned tasks.
textbf Continual Learning Framework Improvement: Task-Specific Parameter Optimization. The framework trains each survival task by allowing the model to optimize the WSI aggregator and the task-specific Cox head jointly by optimizing the loss function, which is defined as LCox = −1/Ne Σ X i:δ i=1 (rˆ i − log X j:Tj≥Ti exp(ˆrj)).
textbf Model Merging Operation Improvement: Adaptive Balancing for Stability. The merging process uses an adaptive balancing factor where the distance between the merged model and the base model remains bounded by the average task norm, which stabiliz[es] the continual merging process across sequential survival tasks.

Abstract

Survival analysis on Whole Slide Images (WSIs) is important in computational pathology for prognosis estimation and treatment planning. However, existing survival models are typically trained independently for each cancer cohort, making continual adaptation computationally expensive for gigapixel-scale WSIs. In this study, we propose MergeSurv, a merging-based continual learning framework for WSI survival analysis. A pathology vision-language foundation model is independently fine-tuned on each task, and the learned parameters are sequentially merged into a unified model without storing previous training data. We further investigate two inference strategies: One-for-All (OFA) and Voting-Expert Aggregation (VEA). Experiments on four TCGA cohorts demonstrate that MergeSurv outperforms naive fine-tuning as well as representative regularization-based and rehearsal-based continual learning methods, while effectively reducing catastrophic forgetting. The results suggest that model merging is a promising direction for scalable and privacy-preserving continual learning in computational pathology.