UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification

arXiv:2508.10954 · cs.LG, cs.AI · Submitted 2025-08-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification".

Jane: Here's the gist of the paper. The gist The proposed UniPrompt-CL method is a medical-oriented Prompt-based Continual Learning (PCL) approach that improves prompt pool design via a minimally expanding unified prompt pool and a new regularization term, achieving better stability–plasticity trade-off with lower computational cost<ref:2508.10954#pg14>.

Tom: Prior PCL methods often underperform or fail to transfer to medical data due to domain bias, institutional constraints, and subtle inter-stage boundaries<ref:2508.10954#pg3>.

Jane: Medical images require different promptlearning strategies than natural images<ref:2508.10954#pg4>.

Tom: Existing PCL methods developed for natural images often learn wide prompt regions to accommodate broad feature differences, which can be inefficient for medical data where fundamental anatomical structures remain consistently important across layers and tasks<ref:2508.10954#pg5>.

Paper summary: Jane: So looking at this paper, "UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification," the authors are proposing a specific way to manage those prompts inside a medical AI model to make continual learning more stable and efficient than what's currently available.

Tom: The title itself points directly to the core idea: using a single inference pass for continual learning in medical image classification, and they claim this method improves accuracy by one to three percentage points over strong baselines.

Lu: It’s about how they design the prompt pool—by unifying it across layers and only expanding it minimally, plus using that new regularization term Ls to guide the learning process toward better stability-plasticity trade-offs.

Meng: For someone deploying this to a real system, the value is in that you get high performance while keeping the inference cost low because you're not running multiple passes or complex pipelines.

Jane: The implication here is that for medical AI where data distribution shifts frequently but compute resources are constrained, this approach offers a practical way to mitigate catastrophic forgetting while maintaining strong feature tracking across the model layers.

Tom: It really boils down to how they managed the prompt design—the unified pool and minimal expansion—which seems to be the key mechanism allowing them to achieve better retention of prior knowledge while still capturing fine-grained features for new medical tasks.

Conclusion: Tom: So we've seen how UniPrompt-CL uses a single inference pass to handle continual learning in medical image classification, and now we gotta talk about what that actually means for us.

Jane: It boils down to this paper, "UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification," written by some folks. The main thing is they tackle the problem of keeping an AI model accurate as it sees new medical data without having to retrain everything from scratch every time.

Tom: Yeah, so they used a unified prompt pool and this minimal expansion strategy to manage the prompts across layers, and that's what keeps things stable. But why does that matter for a doctor or a hospital?

Jane: Because in medicine, you can't just throw all your past patient data into one giant pile anymore because of privacy rules. This method lets the AI adapt to new hospitals or new scanner types while still remembering what it learned before without forgetting the old stuff.

Tom: Exactly. It’s about efficiency here too—they use just one backbone and one inference pass, which is way faster than running a whole pipeline every time you want to update something. Meng, from a deployment standpoint, how does that single pass translate to actual running costs?

Meng: Well, it means less compute needed per update cycle. If I'm shipping this to a clinic where the machine has limited GPU power for retraining, keeping the complexity low with just one pass is actually pretty good for deployment. But we still have to worry about how those prompts are organized across different medical domains.

Jane: That’s a fair point about the cost, Meng. The paper shows they found that this prompt design helps them keep accuracy up—improving it by one to three percentage points on tests like diabetic retinopathy without needing massive training overhead.

Tom: So, the main implication is that for medical AI, we don't have to sacrifice stability just for a little bit of extra accuracy or speed in every single update. It seems the prompt structure itself is smarter than just throwing more layers or more backbones at it.

Jane: It really suggests that when you design these models, especially for specialized fields like medicine, how you manage those internal connections—those prompts—is just as important as the size of the backbone itself.

Tom: Right. So if this prompt strategy works so well here, what’s next? Are they just going to use this for skin cancer and diabetic retinopathy?

Jane: They're looking to see if it holds up on more complex things, like different types of imaging modalities—maybe moving from just standard images to things like CT or MRI scans. We gotta keep asking how robust these prompt designs are across truly different data types.

Gyutae Oh, Jitae Shin

Department of Electrical and Computer Engineering, Sungkyunkwan University

cs.LG, cs.AI

Submitted: 2025-08-14

Updated: 2026-10-21

Comments: Accepted to WACV 2027. 18 pages, including supplementary material

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The proposed UniPrompt-CL method is a medical-oriented Prompt-based Continual Learning (PCL) approach that improves prompt pool design via a minimally expanding unified prompt pool and a new

Key concepts

Unified Prompt Pool: Instead of separate prompt pools for each layer, UniPrompt-CL shares one pool across all transformer layers. This maximizes the use of limited prompt capacity by ensuring that the same set of learned prompts is available at every part of the model, promoting efficient feature tracking throughout the backbone.
Minimal Prompt Expansion: When adapting to a new medical domain, UniPrompt-CL expands the prompt set by adding only 20% more prompts. This controlled expansion ensures that new information is captured without overwhelming the system or increasing computational cost unnecessarily, leading to better adaptation.
Consistency-Enforcing Regularization: A novel regularization term is added to the training loss. This term encourages stable yet adaptive learning, helping the model retain knowledge from previous stages while still being plastic enough to learn new domain features effectively, balancing stability and plasticity.

Terminology

Continual Learning — 3× in this paper · explained in 3 episodes
Continual Learning (CL) — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes

Summary

The gist The proposed UniPrompt-CL method is a medical-oriented Prompt-based Continual Learning (PCL) approach that improves prompt pool design via a minimally expanding unified prompt pool and a new regularization term, achieving better stability–plasticity trade-off with lower computational cost<ref:2508.10954#pg14>.

Motivation

Prior PCL methods often underperform or fail to transfer to medical data due to domain bias, institutional constraints, and subtle inter-stage boundaries<ref:2508.10954#pg3>. Medical images require different promptlearning strategies than natural images<ref:2508.10954#pg4>. Existing PCL methods developed for natural images often learn wide prompt regions to accommodate broad feature differences, which can be inefficient for medical data where fundamental anatomical structures remain consistently important across layers and tasks<ref:2508.10954#pg5>. Furthermore, with layer-wise independent prompt pools, similar low-level information can be repeatedly encoded at multiple layers, resulting in cross-layer redundancy and inefficient use of limited prompt capacity<ref:2508.10954#pg4>.

Proposed Method Components

UniPrompt-CL introduces three key components to improve prompt pool design for efficient and robust continual learning<ref:2508.10954#pg8>. These components are:

A unified prompt pool with minimal expansion<ref:2508.10954#pg8>. This pool is shared across all layers, promoting efficient prompt utilization and maximizing fine-grained feature tracking throughout the hierarchical backbone<ref:2508.10954#pg9>.
A minimal prompt expansion strategy<ref:2508.10954#pg8>. When transitioning to the next CL stage, the prompt set is expanded by adding 20% of the original Lp prompts, and a formulation is introduced to ensure that the added prompts more effectively capture new information<ref:2508.10954#pg8>.
A consistency-enforcing regularization term<ref:2508.10954#pg8>. This novel regularization term is introduced to promote stable yet adaptive learning, achieving a better stability–plasticity trade-off than prior methods<ref:2508.10954#pg8>.

Architectural Design and Training Procedure

The method utilizes a single backbone and a single forward pass, which substantially reduces computational cost while achieving superior performance over prior state-of-the-art (SOTA) methods that rely on multiple backbones or repeated ViT inferences<ref:2508.10954#pg8>. The overall architecture integrates an enhanced prompt pool where the [CLS] token serves as a query at each layer, and these layer-wise queries are centrally managed through the prompt pool integration module<ref:2508.10954#pg7>. The training procedure involves:

Initializing the unified prompt pool K(0) and P(0)<ref:2508.10954#pg8>
For subsequent stages, freezing the existing prompt weights corresponding to previous stages<ref:2508.10954#pg8>
Introducing new prompts by adding 20% of the original Lp prompts and refining them using a loss function involving a regularization term Ls<ref:2508.10954#pg8>.

Experimental Results and Validation

Experiments across diabetic retinopathy and skin cancer datasets demonstrate the effectiveness of UniPrompt-CL<ref:2508.10954#pg7>. The method consistently achieves the highest accuracy and F1-score, with UniPrompt-CL† achieving 0.849±0.011 Acc on DRD<ref:2508.10954#pg7>. Furthermore, the Cost-Adjusted Retained Accuracy (CARA) metric shows that our approach exceeds all SOTA methods, demonstrating the best performance to GFLOPs efficiency<ref:2508.10954#pg7>. The results indicate that the more compact prompt distribution observed in Fig. 2.a is consistent with this design choice and aligns with the improvements in accuracy and F1-Score reported in Table 1<ref:2508.10954#pg7>. External validation on skin cancer datasets showed that our method achieves the highest AvgACC on the skin cancer benchmarks, indicating that the proposed framework can transfer beyond DR and remains effective on a distinct disease and imaging domain<ref:2508.10954#pg7>.

Conclusion

UniPrompt-CL combines a strong foundation backbone with efficient prompt management to mitigate catastrophic forgetting in medical AI<ref:2508.10954#pg7>. The method improves AvgACC by 1–3 percentage points over strong baselines, maintains a favorable efficiencyperformance trade-off by reducing inference cost even when GPLOPs are taken into account, and attains these gains with a single ViT inference–avoiding the multiple passes or complex pipelines required by prior PCL approaches<ref:2508.10954#pg7>. The findings underscore domain-specific considerations, showing that prompt learning behaves differently than in natural images, and our integrated prompt design better preserves prior knowledge while capturing fine-grained features, leading to robust and stable performance<ref:2508.10954#pg7>. Future work plans include extending UniPromptCL to additional modalities like CT, MRI, pathology for broader validation<ref:2508.10954#pg7>.

How it works

The prompt computation is reformulated as follows: phi l = sum m=1 to Lp gamma (q l, k m) p m, where q l is the [CLS] token embedding of each layer and k m and p m are the prompt key and value vectors respectively<ref:2508.10954#pg7>. The resulting prompt phi l is then unified with the layer transformation x l+1 = f l(x l, phi l)<ref:2508.10954#pg7>. The Few Prompt Expansion strategy involves expanding the prompt set by adding 20% of the original Lp prompts and refining them using a loss function Ls defined in Eq. (6)<ref:2508.10954#pg8>. The overall loss function is defined as Ltotal = LCE + lambda reg Ls, where lambda is a hyperparameter set to 0.001 in this study<ref:2508.10954#pg8>. The training procedure follows Algorithm 1, which details the fixed per-stage prompt expansion<ref:2508.10954#pg8>.

Key Metrics

The paper evaluates performance using several metrics to quantify stability and efficiency<ref:2508.10954#pg7>. These include AvgACC, BWT, and AvgF for standard CL metrics<ref:2508.10954#pg7>, and the Cost-Adjusted Retained Accuracy (CARA) which combines performance (AvgACC) and stability (retention = 1 − AvgF) into a single retained performance normalized by GFLOPs<ref:2508.10954#pg7>. A higher CARA indicates more stable and efficient per unit GFLOPs<ref:2508.10954#pg7>. The best results in Table 3 are highlighted in bold, showing UniPrompt-CL† achieving the highest CARA of 0.116<ref:2508.10954#pg7>.

Model Selection Rationale

The authors adopted the DINOv2-base (86.6M parameters) backbone for its balanced trade-off between representational capacity and computational efficiency<ref:2508.10954#pg8>. This choice was made because parameter count grows more than twofold at each step, leading to significantly higher complexity<ref:2508.10954#pg8>. The incorporation of Dino-V2 improves both overall accuracy and robustness against forgetting compared to the original OS-Prompt++ without backbone enhancement<ref:2508.10954#pg8>. This suggests that a stronger backbone provides generalization benefits analogous to few-shot and zero-shot learning, even within PCL settings<ref:2508.10954#pg8>. The selection of the 20% expansion ratio was based on empirical findings showing that increasing from 10% to 20% yielded clear performance improvements, whereas further expanding to 30% increased the parameter count without corresponding gains and even caused slight performance degradation<ref:2508.10954#pg8>. The best results in Table 7 are presented for these expansion ratios<ref:2508.10954#pg8>.

Ablation Study Insights

The ablation study highlights the importance of a good backbone in PCL, showing that our method’s superior performance cannot be attributed to the backbone alone<ref:2508.10954#pg8>. Comparing the original OS-Prompt++ with its DinoV2 variant, we observe that the incorporation of Dino-V2 improves both overall accuracy and robustness against forgetting<ref:2508.10954#pg8>.

Improvements for AI systems

Prompt Pool Consolidation: The system can maintain a unified prompt pool across all layers, which promotes efficient prompt utilization and maximizes fine-grained feature tracking throughout the hierarchical backbone, leading to a more compact and discriminative representation compared to layer-wise independent pools.
Minimal Prompt Expansion: The AI system can adapt to new medical domains by introducing only a small subset of new prompts, as the method introduces a unified prompt pool with minimal expansion and expands only a small subset of prompts to mitigate redundancy and alleviate catastrophic forgetting.
Computational Efficiency: The improved system can achieve superior performance while maintaining lower computational cost by utilizing a single backbone and a single forward pass, which is explicitly stated to avoid the need for two ViT backbones or repeated inference.
Robustness to Domain Shifts: By tailoring prompt learning strategies to medical data characteristics, the system can better handle subtle variations in color and lesion details, leading to improved AvgACC by 1–3 percentage points across domain-incremental learning settings.

Abstract

Medical AI systems must remain reliable as data distributions shift across hospitals, imaging devices, and patient populations, yet privacy constraints often prevent storing or revisiting past patient data. We present UniPrompt-CL, a rehearsal-free domain-incremental learning framework for medical image classification using a single backbone and a single inference. Rather than relying on multiple backbones, repeated ViT inferences, or independent layer-wise prompt pools, UniPrompt-CL shares a unified prompt pool across transformer layers and expands it with only a few prompts per new domain. This reduces cross-layer redundancy while enabling efficient adaptation. On diabetic retinopathy, UniPrompt-CL improves average accuracy by 2.0 percentage points and substantially reduces forgetting relative to the strongest backbone-matched baseline. On skin cancer, it achieves competitive accuracy at the lowest training cost among DINOv2-B configurations. A controlled comparison varying only pool organization shows that the benefit arises from retention rather than learning each new domain: it is largest on the medical benchmarks, smaller on OfficeHome, and reverses on a heterogeneous six-stage DomainNet stream. These results suggest that cross-layer prompt sharing with lightweight expansion is practical for medical models requiring repeated retraining under limited on-site compute, while identifying the domain shifts under which it helps.