Pathology foundation models for training-free case retrieval in salivary gland tumors

arXiv:2610.22087 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Pathology foundation models for training-free case retrieval in salivary gland tumors".

Jane: The gist: Content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're moving on to the next part of this discussion about "Pathology foundation models for training-free case retrieval in salivary gland tumors." We covered what it is and how they set up the search engine.

Jane: So we looked at the title and authors, but let’s dig into what that actually means for a clinician or a researcher working with tissue archives.

Tom: The authors are from institutions like Ludwig-Maximilians-Universität München, Charité-Universitätsmedizin Berlin, and BIFOLD-Berlin. That gives them deep roots in pathology research.

Lu: Their background is very solid because they are coming from places where microscopy is the foundation of diagnosis > Page one #pg1 <ref:2610.22087#pg1>

Jane: So when they talk about training-free retrieval, what’s the simple explanation for that?

Tom: It means the foundation models don't need to be retrained on specific cancer datasets to perform this retrieval task; they learn general representations from massive corpora first.

Meng: But if they aren't trained on tumor data, how do we expect them to recognize a rare tumor type when you query for it?

Jane: They rely on the fact that the foundation models learned general visual patterns from millions of slides, which should include features relevant to any morphology > Page two #pg2 <ref:2610.22087#pg1>

Tom: The paper then uses these pre-trained Vision Transformers, like Atlas2 and UNI2h, as frozen encoders.

Lu: By freezing them means we are relying entirely on their existing ability to generate meaningful representations of tissue structures without changing the model's weights during the retrieval step > Page three #pg3 <ref:2610.22087#pg1>

Jane: So, it’s not teaching the model *what* a tumor looks like; it’s just letting it tell us *how* different visual patterns are structured in general.

Tom: Exactly. The paper then takes these patch-level features and feeds them into the ABMIL model to create a representation for the whole slide image.

Meng: That sounds like they're building a whole new diagnostic tool on top of existing, massive AI models rather than training something from scratch?

Lu: That’s right. It’s leveraging the pre-existing knowledge base of these large models and fine-tuning it just enough to perform this retrieval task > Page three #pg3 <ref:2610.22087#pg1>

Jane: So they are essentially using a very sophisticated pattern recognition engine for searching through their archive.

The paper's summary: Tom: Now we’re getting into the actual details of how they executed this study and what the results actually showed in "Pathology foundation models for training-free case retrieval in salivary gland tumors."

Jane: Can you walk us through the setup—the data preparation, because that’s where a lot of technical detail gets confusing.

Tom: They used a test cohort of nine hundred seventy-four hematoxylin and eosin WSIs from salivary gland tumors for the main test set, plus fifteen non-tumor cases for training.

Meng: So they're looking at a relatively small sample size compared to what you’d expect in a comprehensive pathology archive?

Jane: They are focusing on this specific cohort for testing, but the methodology details how they prepared everything else.

Tom: All scans were acquired at a native magnification of forty times with a resolution of zero point two five µm/pixel using Leica Aperio GT four hundred fifty DX scanners > Page one #pg1

Lu: They tiled the tissue into nonoverlapping patches, and these patches were size five hundred twelve by five hundred twelve pixels at two thousand times magnification covering two hundred fifty-six by two hundred fifty-six micrometers of tissue > Page three #pg3 <ref:2610.22087#pg1>

Jane: And then they extracted features using those frozen models, which resulted in a final layer-normalized CLS embedding vector z in R d > Page one #pg1 <ref:2610.22087#pg1>

Tom: They compared four foundation models: Atlas2, Atlas2-B, Atlas2-S, and UNI2h. The dimension of the feature vectors varied from three hundred eighty-four for Atlas2-S up to two thousand forty-eight for Atlas2 > Page three #pg3

Meng: So they’re comparing models with vastly different feature sizes; that sounds like a lot of overhead in terms of what you’d need to store.

Jane: They trained the ABMIL model using patch-level features from both tumor positive and negative slides to learn those representations > Page one #pg1 <ref:2610.22087#pg1>

Tom: The bag level representation M is formed by taking the attention-weighted average of all those patch embeddings, and then passing that through a single linear classifier w to get the bag-level prediction > Page three #pg3 <ref:2610.22087#pg1>

Lu: They also optimized auxiliary losses, including an instance alignment loss and one exploiting the fact that every patch of a tumorfree case is a confirmed negative instance > Page one #pg1 <ref:2610.22087#pg1>

Jane: So they are ensuring the system learns to distinguish between good and bad cases during training, which is important for accuracy > Page one #pg1 <ref:2610.22087#pg1>

The paper's improvements: Tom: Let’s talk about what they actually improved upon in this research. What were the key advantages of this approach?

Jane: They are showing that content-based retrieval can facilitate rare cancer diagnostics without needing any task-specific training or manual annotation.

Tom: That means we can query a digitized archive using a region of interest from a whole slide image, and retrieve morphologically similar cases along with their ICD-O codes and corresponding WSIs > Page one #pg1 <ref:2610.22087#pg1,and retrieve morphologically similar cases along with their ICD-O codes and>

Lu: This is the main benefit: making departmental archives searchable with pathology foundation models, providing a low barrier route to case-based differential diagnosis > Page two #pg2 <ref:2610.22087#pg1>

Meng: So this is about democratizing access to expertise, right? Letting anyone with a computer and these models start accessing that collective experience without needing years of training >

Jane: Exactly. The study showed that pretrained foundation models can extend an individual pathologist's frame of reference to that of the department > Page one #pg1 <ref:2610.22087#pg1>

Tom: And when we look at the results, Atlas2 demonstrated a significantly higher AUC of zero point seven five four than its distilled counterparts, Atlas2-B and Atlas2-B (zero point six seven nine) and Atlas2-S (zero point six five three) > Page one #pg1

Lu: That difference in performance is significant with p values of zero point zero two zero and p = zero point zero zero three when comparing the models > Page one #pg1

Jane: So, to put it simply, the paper proves that these foundation models work effectively for this purpose without needing any extra task-specific training or manual annotation > Page one #pg1 <ref:2610.22087#pg1>

Tom: That’s a strong result. It shows that using existing AI knowledge is already quite effective for complex visual tasks > Page one #pg1 <ref:2610.22087#pg1>

Meng: But the limitations are clear, though. What stops this from being a production system right now?

Jane: The automatic sourcing through ABMIL didn't show any relevant difference in retrieval accuracy compared to when a pathologist selected queries manually > Page two #pg2 <ref:2610.22087#pg1>

Conclusion: Tom: So we’re wrapping up the discussion on this paper, "Pathology foundation models for training-free case retrieval in salivary gland tumors." We’ve seen how the whole process works from data prep to the final ranking score.

Jane: In short, it offers a way for pathologists to search their archives using their own visual input and these powerful foundation models without needing custom training or annotation.

Tom: The paper demonstrates that pretrained foundation models can extend an individual pathologist's frame of reference to the department's collective experience > Page one #pg1 <ref:2610.22087#pg1>

Lu: This is a tool for making departmental archives searchable with pathology foundation models, providing a low barrier route to case-based differential diagnosis > Page two #pg2 <ref:2610.22087#pg1>

Meng: It’s interesting, but we still need to focus on making the system work reliably in a production environment > Page one #pg1 <ref:2610.22087#pg1>

Jane: We should keep an eye on how this scales and if it improves overall diagnostic workflows for pathologists > Page one #pg1 <ref:2610.22087#pg1>

Maximilian J. Roesel, Niklas Kiermeyer, Philipp Keyl, Frederick Klauschen

Institute of Pathology, Ludwig-Maximilians-Universität München · German Cancer Consortium (DKTK) · German Cancer Research Center (DKFZ) · Institute of Pathology, Charité-Universitätsmedizin Berlin · BIFOLD-Berlin Institute for the Foundations of Learning and Data

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist: Content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training.

Key concepts

Foundation Models (FMs): These are large AI models pre-trained on massive datasets of whole slide images. They learn general visual representations of tissue morphology, allowing them to understand the visual features of pathology slides without needing specific training for every task.
Content-Based Retrieval: Instead of searching by a specific label, this method searches based on the visual content. A user selects an area (ROI) on a new slide; the system converts that area into a numerical description and finds archived cases whose descriptions are visually similar.
ABMIL Model: This is a gated model used to automatically select the most important region of interest on a slide. It helps identify the most diagnostically relevant, spatially contiguous area of tissue, making it easier for pathologists to focus their search.
Content-Based Retrieval Engine: The system uses vectors (numerical representations) generated by FMs to compare queries against archived cases. It ranks matches using a modified Chamfer similarity score, ensuring that the retrieved cases are truly morphologically comparable.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes

Summary

The gist: Content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training.

Introduction and Motivation

Morphology remains the foundation of microscopic pathology, requiring years of training to build an internal frame of reference against which unseen cases are compared <ref:2610.22087#pg4> Mastering its interpretation requires years of training, building an internal frame of reference against which unseen cases are compared <ref:2610.22087#pg6>. While any individual pathologist encounters a given entity only occasionally, a department accumulates such cases over years <ref:2610.22087#pg7>. These institutional archives can become a valuable external frame of reference, yet searching through these archives is challenging due to the diverse morphology and size of WSIs <ref:2610.22087#pg4>. Digitization has made histomorphology accessible to artificial intelligence (AI), which is advancing from research toward routine diagnostics <ref:2610.22087#pg5>. Foundation models (FMs) use self-supervised pre-training on large corpora of slides to learn general-purpose representations that can be used across downstream tasks <ref:2610.22087#pg6>. Content-based retrieval avoids this by returning cases rather than labels <ref:2610.22087#pg6>.

Methodology and Data Preparation

The study utilized a cohort of 137 salivary gland tumor cases, totaling 974 hematoxylin and eosin (H&E) WSIs, for the test cohort <ref:2610.22087#pg7>. For training the ABMIL model, an additional 15 non-tumor cases were digitized, totaling 123 H&E WSIs <ref:2610.22087#pg7>. All scans were acquired at a native magnification of 40× with a resolution of 0.25 µm/pixel (MPP) using Leica Aperio GT 450 DX scanners <ref:2610.22087#pg7>. Every WSI was screened for artifacts using GrandQC <ref:2610.22087#pg8>. Tissue was tiled into nonoverlapping 512×512-pixel patches at 20× magnification covering 256×256 µm of tissue (0.5 MPP) using PathoPatcher The patch size was chosen to balance morphological detail against index size. Patch-level features were extracted with four pathology FMs, all Vision Transformers (ViT) pretrained with the DINOv2 self-supervised framework and used here as frozen encoders. The resulting patch descriptor is the final layer-normalized CLS embedding z ∈ R d.

Foundation Models and ABMIL Training

Four pathology foundation models were compared: Atlas2, Atlas2-B, Atlas2-S, and UNI2h. The feature vector dimensions (d) differed by FM with d = 384 for Atlas2-S, d = 768 for Atlas2-B, d = 1536 for UNI2h, and d = 2048 for Atlas2. The gated ABMIL model was trained using patch-level features from tumor positive and negative slides. A bag is defined as a matrix H ∈ R(n×d) where n is the number of patches for that slide. Patch-level attention weights are computed using the gated attention formulation. The resulting attention-weighted average of patch embeddings forms the bag-level representation M, which is passed through a single linear classifier w to produce the bag-level prediction. Auxiliary losses are optimized, including an instance-alignment loss and a loss exploiting the fact that every patch of a tumorfree case is a confirmed negative instance.

Retrieval Engine and Evaluation

The database indices were built per FM using FAISS (25) where vectors are L2-normalized to unit length so that inner-product equals cosine similarity. A query region is divided into non-overlapping 512 × 512-pixel patches and embedded by an FM, yielding n descriptors. If the pairwise cosine similarity is above 0.8 the query is treated as homogeneous and all descriptors are averaged into a single re-normalized query vector. Retrieved patches are grouped by their source case, and each candidate is ranked by a sparse Chamfer score computed only over the patches retrieved. The final ranking score is the sum of the forward and the adjusted backward term. Evaluation compared FMs via leave-one-out cross-validation across query-source model and index model The study showed that Atlas2 demonstrated a significantly higher AUC (0.754) than its distilled counterparts Atlas2-B (0.679, p = 0.020) and Atlas2-S (0.653, p = 0.003)

Discussion and Conclusion

Content-based retrieval with off-the-shelf pathology FMs identified the correct diagnosis among the top five candidates in up to 81.3% of cases, without any task-specific training or manual annotation. The approach makes departmental archives searchable with pathology foundation models, providing a low-barrier route to case-based differential diagnosis and extending the individual frame of reference to the collective experience of the department. The study shows that pretrained FMs, without further training, can extend an individual pathologist's frame of reference to that of the department. Future work will test whether the tool measurably improves pathologists' performance and how the approach scales to larger cohorts.

--- Page 1 ---

The gist: Content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training.

How it works

The search engine allows users to select a region of interest (ROI) either automatically, using the ABMIL model that identifies the most diagnostically relevant, spatially contiguous region, or manually by selecting a rectangular ROI of arbitrary size from a WSI. This ROI is divided into non-overlapping 512×512-pixel patches and each patch is converted by an FM into a numerical representation that captures its visual features. If the patches within the region are highly similar to one another, the region is treated as homogeneous, and the representations are combined into a single representative query. If the patches are dissimilar, they are grouped into subsets of visually similar tissue, and each subset contributes its own representative query.

Retrieval Process

Each query is then compared against the representations of previously diagnosed archive cases, extracted in advance using the same FM. Matches are scored using our modified Chamfer similarity, returning each match's ICD-O diagnosis alongside its WSI. The final ranking score is the sum of the forward and the adjusted backward term.

Performance Metrics

The evaluation compared FMs via leave-one-out cross-validation across query-source model and index model. Macro-averaged recall at Top-1/3/5 was 62.5%/76.7%/81.3% for Atlas2, 50.2%/66.6%/72.9% for UNI2h, 52.3%/71.5%/77.6% for Atlas2-B, and 54.4%/65.2%/73.7% for Atlas2-S without any statistically significant difference between the models at any depth. The AUC was calculated based on the Top-5 retrieved cases.

Limitations and Future Work

The performance of automated sourcing through ABMIL did not yield a relevant difference in retrieval accuracy compared to pathologist-selected queries. The lower performance of the automatic WSI-to-WSI mode compared with pathologist-selected queries indicates that it does not identify the most relevant query regions, most likely due to limited training data.

Improvements for AI systems

The system can perform content-based retrieval with current pathology foundation models facilitates rare cancer diagnostics without any task-specific training, allowing pathologists to query a digitized archive using a region of interest (ROI) from a whole slide image (WSI) and retrieve morphologically similar cases, along with their ICD-O codes and corresponding WSIs.
The improved system can handle queries where the pathologist selects an ROI manually or uses an automated method, such as the gated attentionbased multiple instance learning model, to identify the most diagnostically relevant, spatially contiguous region from a WSI.
The system can effectively reduce redundancy and manage complexity by treating regions as either homogeneous or heterogeneous: If the patches within the region are highly similar to one another, the region is treated as homogeneous, and the representations are combined into a single representative query.
The retrieval mechanism utilizes a modified Chamfer similarity score to rank retrieved cases, ensuring that nearest-neighbor matching under Euclidean distance and under cosine similarity is equivalent, which provides a robust measure of case comparability.
The system can offer comparative diagnostics by providing performance metrics such as Top-1/3/5 retrieval was 62.5%/76.7%/81.3% for Atlas2, enabling pathologists to assess the reliability of different foundation models (Atlas2, Atlas2-B, Atlas2-S, and UNI2h) without requiring additional training or manual annotation.
The system can provide a mechanism for case-based differential diagnosis and extending the individual frame of reference to the collective experience of the department by allowing users to inspect at their original location within the slide which verifies that the retrieved tissue is genuinely comparable.