GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI

arXiv:2503.04325 · eess.IV, cs.CV · Submitted 2025-03-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI".

Jane: Here's the gist of the paper. The gist The GBT-SAM framework adapts the Segment Anything Model (SAM) to multi-parametric MRI (mp-MRI) data by combining a restructured patch embedding layer, parameter-efficient fine-tuning with LoRA blocks, and a Depth-Condition block to achieve state-of-the art performance on brain tumor segmentation while maintaining computational efficiency.

Tom: <ref:2503.04325#pg2> How it works The GBT-SAM framework addresses limitations in existing segmentation methods by leveraging all four mp-MRI modalities and modeling inter-slice relationships within a parameter-efficient structure<ref:2503.04325#pg7>.

Jane: The model utilizes a two-step training protocol to ensure efficient adaptation of SAM to the task of brain tumor segmentation<ref:2503.04325#pg2> The pipeline involves several key modifications: 1.

Tom: Multi-modal Adaptation for mp-MRI: The patch embedding layer is modified to accept 4-channel inputs instead of the standard 3, allowing it to process all four modalities (T1, T2, T1c, and T2-FLAIR) jointly<ref:2503.04325#pg7>.

Paper summary: Tom: So we're looking at this paper called "GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI". Basically, the authors are trying to tackle the problem of segmenting gliomas, which are these aggressive brain tumors. They note that doing this manually is slow and inconsistent, so they're using deep learning to make it better and faster.

Jane: What's the main thing they claim about this GBT-SAM framework? It seems like they are taking a very large vision model called Segment Anything Model, or SAM, and adapting it specifically for volumetric mp-MRI data. The key idea is making it parameter efficient while still getting good results.

Lu: Right, that's the core concept. They say GBT-SAM reduces the input complexity by selecting fewer than two point six percent of slices per scan while incorporating all four MRI modalities—T1, T2, T1c, and T2-FLAIR—to preserve essential tumor information with minimal computational cost three fifteen.

Meng: Minimal computational cost is what I care about most. When we ship models to production, parameter efficiency matters because of the memory footprint and the inference time. How much training does this framework actually require compared to just fine-tuning a standard UNet?

Tom: Well, they claim their model ends up with just six point five million trainable parameters, which is the lowest among SAM-based approaches <ref:2503.04325#pg2>. That sounds promising for deployment because it keeps the size small while still achieving a high performance metric on the BraTS Adult Glioma dataset.

Jane: So they're not just using one modality or some basic setup; they are specifically addressing the limitations of existing methods that often fail to exploit those inter-slice contextual features in mp-MRI <ref:2503.04325#pg2>. That's a big deal for capturing how tumors spread across the whole brain.

Lalam: I see they focus on using all four sequences, T1c and T2-FLAIR along with T1 and T2, because each sequence gives complementary information about tumor structure and pathology three fifteen <ref:2503.04325#pg2,complementary information about tumor structure and pathology>. That's what makes using all of them crucial for an accurate delineation <ref:2503.04325#pg2>.

Tom: Exactly. And they introduce a depth-aware module in the ViT blocks to explicitly model those inter-slice correlations, which is their way of capturing that context without just throwing massive computational complexity at it <ref:2503.04325#pg7>.

Meng: Modeling inter-slice relationships sounds complex to implement when you’re trying to keep parameters low. What does that depth-aware block actually do in practice? Does it just add a lot of computation, or is it smarter?

Lu: It operates by unfolding the volumetric data along the slice dimension, applying layer normalization, and passing that through a Multi-Layer Perceptron or MLP to learn dependencies across those slices <ref:2503.04325#pg2>. It's a specific way to learn those slice-to-slice links within the encoder structure of the model.

Jane: So they are using this depth information to bridge the gap between how standard image models process slices independently and what radiologists rely on—the continuity of the tumor <ref:2503.04325#pg2>.

Tom: And to make it trainable efficiently, they use a two-step fine-tuning strategy involving LoRA blocks within the ViT modules <ref:2503.04325#pg7>. That's how they specialize the large SAM architecture without needing to update all those billions of original parameters <ref:2503.04325#pg2>.

Lalam: So, essentially, it’s adapting a massive model using targeted updates on top of a multi-modal input and some specific depth conditioning mechanisms <ref:2503.04325#pg7>. It sounds like they are hitting the right spot for efficiency while trying to get high performance.

Meng: From an engineering standpoint, I'm interested in that two-step fine-tuning protocol. Does it make the training process more stable, or does it introduce new bottlenecks when we try to scale up the data volume?

Conclusion: Tom: So we’ve covered the GBT-SAM framework—a parameter-efficient deep learning method that adapts SAM for mp-MRI by using four modalities, a depth-aware block, and LoRA blocks <ref:2503.04325#pg7>. The big win they show is achieving a Dice Score of ninety-three point five four on the BraTS Adult Glioma dataset <ref:2503.04325#pg1>.

Jane: And it's not just good on that one dataset, either. They demonstrated robust performance across other datasets like Meningioma, Pediatric Glioma, and Sub-Saharan Glioma <ref:2503.04325#pg1>. This suggests the method is generalizable beyond just the initial training data <ref:2503.04325#pg1>.

Lu: The authors found that using all four mp-MRI modalities yields the best generalization capability across all not seen brain tumor domains <ref:2503.04325#pg1>. They also found that a specific slice selection strategy, using slices with delta equal to one gave them the best performance across those different datasets <ref:2503.04325#pg1>.

Meng: I think what this means for deployment is that if we can achieve competitive results with just six point five million parameters, that’s a huge win for clinical tools. It suggests we might be able to deploy sophisticated segmentation models on devices with tighter computational constraints than what was previously feasible <ref:2503.04325#pg1>.

Tom: That’s right, the trade-off they highlight is between accuracy and computational cost <ref:2503.04325#pg1>. They managed to get state-of-the-art performance while keeping the parameter count extremely low compared to other SAM adaptations <ref:2503.04325#pg1>.

Jane: So, in simple terms, this paper proposes a way to use deep learning models like SAM more smartly for brain tumor segmentation by paying close attention to the context between slices and using only the necessary parts of the data efficiently <ref:2503.04325#pg1>. It shows that combining multi-modal input with depth modeling is a practical path forward <ref:2503.04325#pg1>.

Lalam: I'm fairly confident in what this paper says about the performance metrics compared to other SAM-based approaches <ref:2503.04325#pg1>. The results are very strong on the BraTS Adult Glioma dataset, and the extension to different tumor types shows good stability <ref:2503.04325#pg1>.

Tom: So, it’s a framework that manages to be both computationally efficient and robust across different types of gliomas using mp-MRI data <ref:2503.04325#pg1>. That's what this GBT-SAM paper is all about.

Cecilia Diana Albeldaa, Roberto Alcover Cousoa, A'lvaro García-Martín, Jesus Bescosa, Marcos Escudero-Viğola

Universidad Autonoma de Madrid · Amazon

eess.IV, cs.CV

Submitted: 2025-03-06

Updated: 2025-12-01

Journal ref: International Journal of Imaging Systems and Technology 36(4), e70389 (2026)

DOI: 10.1002/ima.70389

Code: https://github.com/vpulab/med-sam-brain

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: The gist The GBT-SAM framework adapts the Segment Anything Model (SAM) to multi-parametric MRI (mp-MRI) data by combining a restructured patch embedding layer, parameter-efficient fine-tuning with

Key concepts

Segment Anything Model (SAM)
SAM is a large vision model designed for zero-shot image segmentation. GBT-SAM adapts this powerful pre-trained model to the specific task of segmenting brain tumors from complex volumetric mp-MRI data, making it highly versatile for different tumor types.
Multi-modal Adaptation
This involves modifying the input layer of the model to handle four different MRI modalities (T1, T2, T1c, and T2-FLAIR) simultaneously. This allows the model to use complementary diagnostic information from all scans at once, improving segmentation accuracy.
Depth-Condition Block
This module is added within the image encoder to explicitly model how different slices relate to each other in a 3D volume. It learns inter-slice correlations by unfolding the data and using a Multi-Layer Perceptron (MLP) to capture these spatial dependencies.
Parameter-Efficient Fine-Tuning (LoRA)
Instead of retraining the entire large model, GBT-SAM uses LoRA blocks within the Vision Transformer (ViT) modules. This technique restricts parameter updates to a low-dimensional subspace, allowing for specialization on brain tumor segmentation while keeping the total number of trainable parameters very low.

Terminology

Summary

The gist The GBT-SAM framework adapts the Segment Anything Model (SAM) to multi-parametric MRI (mp-MRI) data by combining a restructured patch embedding layer, parameter-efficient fine-tuning with LoRA blocks, and a Depth-Condition block to achieve state-of-the art performance on brain tumor segmentation while maintaining computational efficiency. <ref:2503.04325#pg2>

How it works

The GBT-SAM framework addresses limitations in existing segmentation methods by leveraging all four mp-MRI modalities and modeling inter-slice relationships within a parameter-efficient structure<ref:2503.04325#pg7>. The model utilizes a two-step training protocol to ensure efficient adaptation of SAM to the task of brain tumor segmentation<ref:2503.04325#pg2>

The pipeline involves several key modifications:

  1. Multi-modal Adaptation for mp-MRI: The patch embedding layer is modified to accept 4-channel inputs instead of the standard 3, allowing it to process all four modalities (T1, T2, T1c, and T2-FLAIR) jointly<ref:2503.04325#pg7>. This ensures that the model receives the same complementary diagnostic information that clinicians routinely assess<ref:2503.04325#pg7>.

  2. Depth-Conditioned Correlation Modelling: A Depth-Condition block is introduced in each ViT block of the image encoder to model interactions between slices<ref:2503.04325#pg2>. This module operates by unfolding the volumetric data along the slice dimension, applying layer normalization, and passing it through a Multi-Layer Perceptron (MLP) to learn dependencies across slices<ref:2503.04325#pg2>.

  3. Parameter-Efficient Domain Adaptation: The model employs a parameter-efficient fine-tuning strategy based on Low-Rank Adaptation (LoRA) blocks within the ViT modules of the image encoder<ref:2503.04325#pg2>. This approach restricts the parameter update to a low-dimensional subspace, preserving the foundational structure of SAM while enabling it to specialize to volumetric brain tumor segmentation with minimal additional computational cost<ref:2503.04325#pg2>.

Training and Optimization

The training process is structured in a two-step protocol designed for efficient adaptation<ref:2503.04325#pg2>.

First Step:

The entire architecture is frozen except for the patch embedding layer, which processes the full mp-MRI input to capture information from all four MRI modalities<ref:2503.04325#pg2>. This step optimizes the patch embedding layer to handle the multi-modal nature of the data before introducing additional trainable parameters<ref:2503.04325#pg2>.

Second Step:

In this step, the patch embedding layer remains trainable, and LoRA blocks are introduced within the ViT modules of the image encoder<ref:2503.04325#pg2>. This second step focuses on fine-tuning the system as a whole, achieving a better combination of the image encoder and the modified patch embedding<ref:2503.04325#pg2>.

Experimental Setup and Results

The study utilizes datasets from the public RSNA-ASNR-MICCAI BraTS 2023 challenge, extending evaluation to four distinct brain tumor domains: Adult Glioma (D1), Meningioma (D2), Pediatric Glioma (D3), and Sub-Saharan Glioma (D4)<ref:2503.04325#pg2>. The primary evaluation metric used is the Dice Score (DS)<ref:2503.04325#pg2>.

The results demonstrate that GBT-SAM achieves a Dice score of 93.02 on the BraTS Adult Glioma dataset (DS 1), which is the training set<ref:2503.04325#pg2>. Furthermore, it achieves superior generalization capability across unseen brain tumor domains, as evidenced by its mean performance on unseen datasets (DS 234)<ref:2503.04325#pg2>.

A comparison of MRI modalities shows that leveraging all four mp-MRI modalities yields the best generalization capability across all not seen brain tumor domains (DS 234), while maintaining strong performance on D1<ref:2503.04325#pg2>. The optimal slice selection strategy for training is found to be using slices with δ = 1, which yields the best performance across all four datasets (D1, D2, D3, and D4)<ref:2503.04325#pg2>.

Conclusion

GBT-SAM represents a step towards efficient and generalizable brain tumor segmentation from mp-MRI within the broader context of imaging systems and technology<ref:2503.04325#pg2>. The framework achieves state-of-the art performance on the BraTS Adult Glioma dataset with a Dice score of 93.02, while maintaining strong generalization to additional tumor domains, including Meningioma, Paediatric Glioma, and Sub-Saharan Glioma<ref:2503.04325#pg2>. A key advantage is the favourable trade-off between accuracy and computational cost<ref:2503.04325#pg2>. By jointly exploiting all four MRI modalities, using LoRA-based adaptation, and training with a slice selection strategy that processes fewer than 2.6% of slices per iteration (4 out of 155), the model reaches competitive performance with fewer than 6.5M trainable parameters<ref:2503.04325#pg2>. The Depth-Condition block further enhances encoder features by modeling inter-slice dependencies, which is particularly important for volumetric medical imaging, without introducing substantial computational overhead<ref:2503.04325#pg2>. This makes GBT-SAM a practical candidate for integration into resource-constrained clinical imaging workflows<ref:2503.04325#pg2>. The method relies on bounding box prompts, which aligns with realistic clinical use but requires user-provided annotations at inference time<ref:2503.04325#pg2>. The modified patch embedding layer is tailored to four specific MRI modalities, which limits direct applicability to datasets with different or fewer imaging channels<ref:2503.04325#pg2>. Additional work is needed to generalize the architecture to variable modality configurations<ref:2503.04325#pg2>. This work represents a step towards efficient and generalizable brain tumor segmentation from mp-MRI within the broader context of imaging systems and technology<ref:2503.04325#pg2>. The work was supported by the Ministerio de Ciencia e Innovación of the Spanish Government (grant PID2021-125051OB-I00) and by the Regional Government of Madrid of Spain (grant TEC 2024/COM-322)<ref:2503.04325#pg2>. The content of this paper is organized as follows: Section 2 reviews the current state of the art in brain tumor segmentation and LVM adaptation<ref:2503.04325#pg2>. Section 3 details our proposed architecture, GBT-SAM, including the multi-modal embedding strategy Depth-Condition block, and the training protocol<ref:2503.04325#pg2>. Section 4 presents the experimental setup, ablation studies and both quantitative and qualitative results<ref:2503.04325#pg2>. Finally, Section 5 summarizes the main findings, discusses limitations, and outlines directions for future work<ref:2503.04325#pg2>. The content of this paper is organized as follows: Section 2 reviews the current state of the art in brain tumor segmentation and LVM adaptation<ref:2503.04325#pg2>. Section 3 details our proposed architecture, GBT-SAM, including the multi-modal embedding strategy Depth-Condition block, and the training protocol<ref:2503.04325#pg2>. Section 4 presents the experimental setup, ablation studies and both quantitative and qualitative results<ref:2503.04325#pg2>. Finally, Section 5 summarizes the main findings, discusses limitations, and outlines directions for future work<ref:2503.04325#pg2>. The content of this paper is organized as follows: Section 2 reviews the current state of the art in brain tumor segmentation and LVM adaptation<ref:2503.04325#pg2>. Section 3 details our proposed architecture, GBT-SAM, including the multi-modal embedding strategy Depth-Condition block, and the training protocol<ref:2503.04325#pg2>. Section 4 presents the experimental setup, ablation studies and both quantitative and qualitative results<ref:2503.04325#pg2>. Finally, Section 5 summarizes the main findings, discusses limitations, and outlines directions for future work<ref:2503.04325#pg2>. The content of this paper is organized as follows: Section 2 reviews the current state of the art in brain tumor segmentation and LVM adaptation<ref:2503.04325#pg2>.

Improvements for AI systems

  1. Bold header: Multi-modal Input Processing

The improved system can process all four mp-MRI modalities (T1, T2, T1c, and T2-FLAIR) jointly by modifying the patch embedding layer to accept a 4-channel input, ensuring the full multi-modal information from mp-MRI is leveraged.

  1. Bold header: Inter-slice Correlation Modeling

The system can capture inter-slice dependencies by introducing a Depth-Condition block in each ViT block of the image encoder, which models interactions between slices to provide depth-aware representations that reflect the interactions learned.

  1. Bold header: Parameter Efficiency and Domain Adaptation

The model can be trained efficiently using a two-step protocol where it begins by training only the patch embedding layer and then proceeds with LoRA blocks [54] introduced within the ViT modules of the image encoder, resulting in only 6.5 M trainable parameters while achieving domain adaptation.

  1. Bold header: Enhanced Generalization Across Tumor Types

The GBT-SAM framework can demonstrate robust generalization by being evaluated on four distinct brain tumor domains (Adult Glioma, Meningioma, Pediatric Glioma, and Sub-Saharan Gliomas), highlighting its potential generalizability and adaptability to diverse clinical settings.

  1. Bold header: Optimal Slice Sampling Strategy

The system can optimize computational efficiency by employing a specific training strategy where it selects slices such that we sample only four slices per volume (approximately 2.6 % of the full stack) at each iteration, with the best performance achieved when using slices with δ = 1.

Abstract

Gliomas are aggressive brain tumors that require accurate imaging-based diagnosis, with segmentation playing a critical role in evaluating morphology and treatment decisions. Manual delineation of gliomas is time-consuming and prone to variability, motivating the use of deep learning to improve consistency and alleviate clinical workload. However, existing methods often fail to fully exploit the information available in multi-parametric MRI (mp-MRI), particularly inter-slice contextual features, and typically require considerable computational resources while lacking robustness across tumor type variations. We present GBT-SAM, a parameter-efficient deep learning framework that adapts the Segment Anything Model (SAM), a large-scale vision model, to volumetric mp-MRI data. GBT-SAM reduces input complexity by selecting fewer than 2.6% of slices per scan while incorporating all four MRI modalities, preserving essential tumor-related information with minimal cost. Furthermore, our model is trained by a two-step fine-tuning strategy that incorporates a depth-aware module to capture inter-slice correlations and lightweight adaptation layers, resulting in just 6.5M trainable parameters, which is the lowest among SAM-based approaches. GBT-SAM achieves a Dice Score of 93.54 on the BraTS Adult Glioma dataset and demonstrates robust performance on Meningioma, Pediatric Glioma, and Sub-Saharan Glioma datasets. These results highlight GBT-SAM's potential as a computationally efficient and domain-robust framework for brain tumor segmentation using mp-MRI. Our code and models are available at https://github.com/vpulab/med-sam-brain.

Sources

Related papers