SAM-DCE: Decoupled Context Enhancement for Prompt-Free Medical Image Segmentation

The gist The proposed SAM-DCE framework introduces a prompt-free medical segmentation method that addresses token uniformity and semantic over-smoothing in Segment Anything Model (SAM) by employing a dual-path module, ML-DCE, to balance local discrimination and global semantics.

Problem Definition

Prior studies have demonstrated that the self-attention mechanism in ViTs essentially functions as a low-pass filter, smoothing input features and reducing their variance. As shown Fig.2, SAM stacks ViT layers repeatedly perform spatial smoothing, leading to homogeneous representations of mask tokens. As a result, the model tends to overemphasize high-frequency, strongly discriminative regions while underrepresenting boundaries and tail classes. Since SAM’s final segmentation results rely heavily on the outputs of deep mask tokens, this imbalance further undermines overall performance, particularly in fine-grained boundaries and low-contrast regions.

Multi-Level Decoupled Class Embedding (ML-DCE)

To address token homogenization, the paper proposes the ML-DCE module, which contains two complementary submodules: MCC and ICC. MCC extracts high-level semantic and boundary information from deep decoder layers, while ICC aggregates spatial details from shallow encoder features. Together, they enhance semantic separability and structural consistency, improving segmentation accuracy, particularly for boundaries and low-frequency classes.

Mask-Conditioned Class Token (MCC)

The MCC module employs cross-attention to decouple and aggregate class semantics from candidate mask tokens. Specifically, the mechanism involves using a set of class-initialized queries Q0 ∈ R 1×C×D, which are broadcast to the batch dimension as Q = expand(Q0, B) ∈ R B×C×D. Each class query corresponds to a foreground category and serves as a guide to attend to the mask tokens generated by the decoder. This operation is formalized as A = softmax (QWQ(KWK) T / sqrt(dk)), TMask agg = A(V WV). The final category-aligned representation is obtained by passing the interaction results through a non-linear feedforward network (MLP), yielding the final category-aligned representation: TMCC = MLPT Magg ∈ R B×C×D.

Image-Conditioned Class Token (ICC)

The ICC module incorporates a set of learnable class queries that attend to encoder-derived image embeddings via cross-attention. This design enables ICC to explicitly aggregate spatial priors, boundary-aware cues and local contextual semantics, thereby constructing class-level representations that are structurally grounded. Formally, given encoder embeddings S ∈ R B×N×D, crossattention produces intermediate outputs H(h) for each head h. The final output of ICC is TICC = MLP T Iagg ∈ R B×C×D.

Residual Fusion and Training Objective

The residual stage introduces a Class-Semantic Enhancement mechanism where the background token tbg is separated from TMask to prevent class semantics from being mixed with background information during fusion. A foreground-enhanced sequence is obtained by performing residual fusion between the deep and shallow class information learned by MCC and ICC and the original class tokens, yielding a foregroundenhanced sequence. Finally, the output Tnew is obtained by concatenating tbg with TfgMask, TMCC + α TMCC + β TICC. The overall loss function Lloss is defined as a weighted combination of the Cross-Entropy loss (LCE) and the Dice similarity loss (LDice)

Experimental Results

SAM-DCE achieves state-of-the-art performance with a Mean Dice of 80.54% and an HD95 of 13.66 mm. Our model substantially outperforms the best SAM-based variant, SAMed, improving the Mean Dice by +2.17% while more than halving the boundary error (13.66 mm vs. 27.81 mm). The ablation study shows that incorporating both MCC and ICC together achieves the best performance with a Mean Dice of 80.54%.

Conclusion

In this paper, we have presented SAM-DCE, a dual-path promptfree decoding framework designed to enhance SAM-based medical image segmentation. ML-DCE explicitly disentangles and refines mask tokens through two complementary modules.

1. Dual-path token generation for enhanced representation: The ML-DCE module balances local discrimination and global consistency by extracting class semantics from mask tokens using MCC and capturing spatial cues from encoder features via ICC. This allows the system to alleviate token uniformity and enhance inter-class separability, leading to improved boundary delineation.

2. Semantic refinement of mask tokens: The MCC module uses class-initialized queries to perform selective attention over decoder-generated mask tokens, yielding a category-aligned representation called TMCC. This process ensures that the model extracts features most relevant to its semantic category, mitigating the effect where self-attention mechanism in ViTs essentially functions as a low-pass filter.

3. Global spatial context integration: The ICC module incorporates learnable class queries that attend to encoder embeddings via cross-attention, effectively integrating spatial priors, boundary-aware cues and local contextual semantics into globally aware class representations. This constructs tokens that are structurally grounded, improving the segmentation of low-frequency categories.

4. Robust loss function for training: The model utilizes a hybrid loss function Lloss = weighted combination of the Cross-Entropy loss (LCE) and the Dice similarity loss (LDice). This formulation leverages the pixel-level classification accuracy of Cross-Entropy with the global structure awareness of the Dice score, providing a more comprehensive and robust supervisory signal for model optimization.

5. Improved performance on medical benchmarks: SAM-DCE achieves state-of-the-art results by substantially improving Mean Dice by +2.17% over SAMed and halving boundary error (13.66 mm vs 27.81 mm), demonstrating superior accuracy in complex tasks like segmenting the Left Kidney (91.82%).

The Segment Anything Model (SAM) is a promising foundation model for medical image segmentation, but its prompt-driven design is not directly suitable for fully automatic multi-class anatomical segmentation. Without manual point, box, or mask prompts, SAM faces a dual over-smoothing problem: deep image features may lose fine local structural evidence, while mask tokens may become semantically ambiguous. We propose SAM-DCE, a decoupled context enhancement framework for prompt-free medical image segmentation. SAM-DCE introduces Spatial Representation Compensation (SRC) to recover anatomical evidence from intermediate ViT features and Semantic Token Calibration (STC) to improve the category consistency of foreground mask tokens. Rather than replacing the pretrained decoder or fusing output masks, SAM-DCE refines the spatial and semantic representations before recombining them through SAM's original token-to-mask mechanism. On the Synapse multi-organ CT segmentation dataset, SAM-DCE achieved a mean Dice score of 83.86% and an HD95 of 16.04 mm, outperforming the representative CNN-, Transformer-, and SAM-based baselines evaluated here. Ablation experiments further showed that SRC and STC provide complementary improvements, supporting the proposed spatial--semantic factorization for prompt-free SAM adaptation.