Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

arXiv:2607.02091 · cs.CV · Submitted 2026-07-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors".

Tom: The gist Breast fibroadenoma (FA) and phyllodes tumor (PT) are fibroepithelial breast lesions with highly overlapping appearances on B-mode ultrasound,

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're now moving into how they actually set up their approach for this "Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors" paper. They show us the framework they used to combine all those different types of information.

Jane: They start by building these modality-specific feature encoders, one for the images, one for the text descriptions, and one for those structured clinical variables.

Meng: I wonder what kind of encoders they chose to pull out these features from so they could get good representations from each source >

Lu: They use DenseNet-based visual encoding for the images because CNNs are strong at modeling local texture and structural features, like Salehi et al. reported with those architectures >

Tom: And for the text, they use something inspired by CLIP to create a text encoding, which is supposed to capture semantic meaning from the written reports.

Jane: So then they take all these different modality-specific features and project them into one unified embedding space using learnable linear transformations and layer normalization to achieve cross-modal alignment >

Meng: That sounds like a standard way to try and make sure the visual data and the text data are speaking the same language before they interact >

Tom: But that’s not all. They introduce this thing called Clinical-guided Adaptive Modulation. This module takes clinical embeddings and uses them to generate parameters called FiLM parameters or adaptive gates >

Jane: This is what makes it clinically guided; it lets the clinical information actually modulate or weigh the visual and textual features directly before they fuse >

Lu: It’s like you have a control panel based on a patient's age or tumor size that you can use to fine-tune how much weight the visual and text features get in the fusion process >

Tom: The modulation mechanism is defined as zmod = z + g(zclin) · (γ ⊙ z + β), where g represents a learnable gating function constrained between zero and one >

Meng: So it’s a way to apply clinical context as a residual reweighting mechanism, which sounds like it could be pretty powerful for filtering out noisy visual data based on known patient characteristics >

Jane: And this modulation is applied independently to both the visual and textual embeddings, resulting in separate modulated versions of each >

Conclusion: Tom: So we've covered the whole thing. We looked at how they built this FAPT-M Dataset, how they used dense vision encoders and CLIP for text, and how they layered clinical information on top with that adaptive modulation.

Jane: What does this all mean when you think about what the authors of "Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors" actually achieved?

Tom: The results under patient-level five-fold cross-validation showed they got an accuracy of seventy-seven point six four percent, an F1-score of seventy-three point three eight percent, and an AUC of eighty-nine point seven four percent.

Meng: That's a solid performance, especially when they compared it against other models like DenseNet, Swin, DeiT, and CLIP >

Lu: They showed that this multimodal framework achieves superior performance in fine-grained breast tumor classification compared to conventional CNNs and Transformers that only look at one modality >

Jane: And the discussion points out that the combination of ultrasound images, diagnostic text descriptions, and structured clinical variables helps them capture subtle differences between those highly similar lesions.

Tom: It suggests that incorporating these different kinds of information makes the model more robust because it incorporates semantic and clinical cues beyond just what you see in an image >

Jane: So for someone listening to the show who only sees a picture, this means that if you’re trying to diagnose something tricky like fibroadenoma versus borderline phyllodes tumor, having those written reports and patient history helps the AI make a better call >

Meng: The authors also mentioned that their framework maintains stable performance even when the class distribution is controlled, which suggests they aren't just getting lucky because of how many examples they have for one category >

Tom: That stability is important; it shows that these multimodal fusion techniques can retain good results even when you try to balance the data between the different tumor types.

Jane: So, in simple terms, this paper proves that by combining image data with text and clinical details, we get a better system for figuring out those tough breast lesions than just looking at the picture alone >

School of Integrated Circuits, Wuxi Vocational College of Science and Technology · School of Advanced Technology, Xi’an Jiaotong-Liverpool University · Shanghai Gem Science and Technology Co., Ltd. · Department of Oncology, Shanghai Medical College, Fudan University · Department of Medical Ultrasound, Fudan University Shanghai Cancer Center

cs.CV

Submitted: 2026-07-02

Updated: 2026-10-09

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: The gist Breast fibroadenoma (FA) and phyllodes tumor (PT) are fibroepithelial breast lesions with highly overlapping appearances on B-mode ultrasound, making benign and borderline PT prone to being

Key concepts

FAPT-M Dataset
A custom dataset of 910 patients containing ultrasound images, clinical data like age and size, and diagnostic text. It is carefully curated to ensure image labels match pathology reports exactly, creating a realistic training environment that reflects real-world diagnostic workflows.
Clinical-Guided Adaptive Modulation
A module that uses structured clinical information to dynamically adjust the visual and text features. It generates parameters based on clinical data to modulate the input embeddings, acting like a clinically controlled reweighting mechanism to enhance feature interaction between modalities.
Transformer-Based Fusion Head
The final part of the model that uses a transformer architecture. It takes processed visual, textual, and clinical features as tokens and uses self-attention to model complex relationships across all three data types simultaneously. This captures deep contextual dependencies for accurate classification.

Terminology

Summary

The gist Breast fibroadenoma (FA) and phyllodes tumor (PT) are fibroepithelial breast lesions with highly overlapping appearances on B-mode ultrasound, making benign and borderline PT prone to being misclassified as FA and complicating preoperative decision-making<ref:2607.02091#pg2>

Dataset Construction

The study constructs the FAPT-M Dataset, a pathology-confirmed multimodal dataset comprising 910 patients with strictly reviewed ultrasound images, structured clinical attributes, and ultrasound diagnostic descriptions<ref:2607.02091#pg4> This dataset better reflects real-world diagnostic workflows<ref:2607.02091#pg4> The categories of all cases are verified, and a one-to-one correspondence between images and pathology is established to avoid label noise interfering with model training<ref:2607.02091#pg4> Structured clinical variables (including age, tumor size, and standardized diagnostic text) are systematically integrated, and standardization is completed in the preprocessing stage to offer standardized input for subsequent multimodal feature fusion<ref:2607.02091#pg4>

Multimodal Framework

The proposed framework integrates DenseNet-based visual encoding, CLIP-inspired text encoding, and lightweight clinical encoding<ref:2607.02091#pg4> It proposes a clinically guided multimodal framework that integrates DenseNet-based visual encoding, CLIP-inspired text encoding, and lightweight clinical encoding<ref:2607.02091#pg4> The modality-specific feature encoders are designed to extract discriminative high-level semantic representations from heterogeneous information sources and map different modalities into a unified embedding space<ref:2607.02091#pg4> These features are then projected into a unified embedding space via learnable linear transformations followed by layer normalization to achieve cross-modal semantic alignment<ref:2607.02091#pg4>

Clinical-Guided Adaptive Modulation

The framework incorporates structured clinical priors into the feature transformation process through a Clinical-guided Adaptive Modulation module<ref:2607.02091#pg4> This module generates FiLM parameters and adaptive gates from the clinical embedding to modulate the image and text features<ref:2607.02091#pg4> The modulation mechanism is defined as zmod = z + g(zclin) · (γ ⊙ z + β), where g(·) represents a learnable gating function constrained within (0, 1)<ref:2607.02091#pg4> This structure can be construed as a clinically controlled residual reweighting mechanism<ref:2607.02091#pg4> The modulation process is independently applied to visual and textual embeddings, resulting in zmodimg and zmodtxt<ref:2607.02091#pg4>

Transformer-Based Fusion Head

The framework introduces a transformer-based fusion head that first constructs token-level cross-modal interactions in a unified embedding space<ref:2607.02091#pg4> A multimodal token sequence T = zmodimg, zmodtxt, zclin is constructed to map heterogeneous features into homogeneous representational units<ref:2607.02091#pg4> This token sequence is subsequently fed into a multi-layer transformer encoder where multi-head self-attention performs global dependency modeling<ref:2607.02091#pg4> The final fused representation is obtained as Tfused = hz fimg, z ftxt, z fclini<ref:2607.02091#pg4> A dual-path fusion strategy is adopted where the raw pathway preserves modality-specific discriminative capacity and the interaction pathway captures contextual dependencies<ref:2607.02091#pg4> The final prediction is obtained by yˆ = Classifier (zfusion)<ref:2607.02091#pg4>

Experimental Results

Under patient-level five-fold cross-validation, the proposed method achieves an accuracy of 77.64%, F1-score of 73.38%, and AUC of 89.74% The proposed model consistently outperforms representative CNN and Transformer baselines such as DenseNet, Swin, DeiT, and CLIP<ref:2607.02091#pg5> In terms of ACC and F1-Score, the proposed model consistently outperforms representative CNN and Transformer baselines such as DenseNet[31], EfficientNetV2[43], Swin[44], and DeiT[47]<ref:2607.02091#pg6> The results demonstrate that the proposed multimodal framework achieves superior performance in fine-grained breast tumor classification compared with conventional CNNs, Transformers, and modality-limited models<ref:2607.02091#pg6> The proposed model still achieves the best overall ACC under this controlled setting [Table 3] and maintains competitive performance for the other subcategories[Table 3]<ref:2607.02091#pg6> Overall, the experimental results demonstrate that the proposed multimodal framework achieves superior performance in fine-grained breast tumor classification compared with conventional CNNs, Transformers, and modality-limited models<ref:2607.02091#pg6>

Discussion

The clinical-conditioned modulation and cross-modal interaction modules enable adaptive feature fusion while preserving modality-specific information<ref:2607.02091#pg4> The combination of ultrasound images, diagnostic text descriptions, and structured clinical variables enhances discriminative representation, especially when imaging features are highly similar across subtypes<ref:2607.02091#pg4> The results suggest that multimodal fusion improves robustness by incorporating semantic and clinical cues beyond visual appearance<ref:2607.02091#pg4> The proposed framework maintains stable performance even when class distribution is controlled, suggesting that the observed performance improvements are not solely due to class distribution bias<ref:2607.02091#pg4> This indicates that the proposed multimodal framework can retain stable performance even when class distribution is controlled<ref:2607.02091#pg4>

Limitations and Future Work

The current study is based on a single-center retrospective dataset, and external validation on multi-center cohorts is required to further evaluate generalization performance<ref:2607.02091#pg4> Additional imaging modalities such as elastography, Doppler, and contrast-enhanced ultrasound may further improve performance in future work<ref:2607.02091#pg4> The current framework focuses on image-level classification and does not incorporate explicit lesion segmentation<ref:2607.02091#pg4> Integrating segmentation or localization may further enhance feature learning<ref:2607.02091#pg4> These findings suggest that multimodal learning can improve diagnostic accuracy in challenging breast tumor differentiation tasks<ref:2607.02091#pg4> The proposed method achieves the best overall performance across most evaluation metrics[Table 2]<ref:2607.02091#pg6> The proposed model achieves the best overall performance and maintains consistent improvements across both FA and PT categories[Table 4]<ref:2607.02091#pg6> The complete multimodal configuration delivers the most robust and comprehensive results across various evaluation metrics<ref:2607.02091#pg6> The proposed framework achieves superior performance in fine-grained lesion classification<ref:2607.02091#pg4> The code is available at GitHub<ref:2607.02091#pg4>

Conclusion

In this paper, we proposed a multimodal deep learning framework for fine-grained classification of FA and PT tumors using ultrasound data, ultrasound diagnostic text, and clinical variables<ref:2607.02091#pg4> By integrating ultrasound images, diagnostic text descriptions, and structured clinical information, the model leverages complementary information from multiple modalities to better capture subtle differences between highly similar lesions<ref:2607.02091#pg4> The proposed bidirectional cross-modal attention and gated modulation mechanisms enable effective semantic alignment and adaptive feature fusion across modalities<ref:2607.02091#pg4> Experimental results demonstrate that the proposed method consistently outperforms several strong baseline models and maintains stable performance across ablation studies and multi-center validation, indicating robustness and generalization<ref:2607.02091#pg4> These findings suggest that multimodal learning can improve diagnostic accuracy in challenging breast tumor differentiation tasks<ref:2607.02091#pg4> The proposed framework achieves superior performance in fine-grained breast tumor classification compared with conventional CNNs, Transformers, and modality-limited models<ref:2607.02091#pg6> The proposed method achieves the best overall performance across most evaluation metrics[Table 2]<ref:2607.02091#pg6> The proposed model achieves the best overall performance and maintains consistent improvements across both FA and PT categories[Table 4]<ref:2607.02091#pg6> The complete multimodal configuration delivers the most robust and comprehensive results across various evaluation metrics<ref:2607.02091#pg6> The proposed framework achieves superior performance in fine-grained lesion classification<ref:2607.

Improvements for AI systems

  1. Bold header: Clinical-Conditioned Adaptive Modulation

Adaptive feature recalibration is achieved by generating feature-wise modulation parameters from clinical embeddings using F(iLM) and applying them via a gated residual structure, which allows the representation space to deform adaptively based on patient-specific priors.

  1. Bold header: Transformer-Based Cross-Modal Fusion Head

A token sequence is constructed from modality representations, where the self-attention mechanism computes correlation weights among tokens, enabling visual representations to be refined according to textual semantics and clinical context.

  1. Bold header: Dual-Path Fusion Strategy

The framework uses a dual-path fusion strategy where the representation is formed by concatenating the raw pathway (zimg, ztxt, zclin) with the interaction pathway (z fimg, z ftxt, z fclini), ensuring that critical local information is retained while benefiting from contextual enhancement.

  1. Bold header: Multimodal Feature Encoding

Independent encoders are used for each modality—DenseNet121 for images, a pretrained language encoder for text, and an MLP for clinical variables—which are then aligned into a shared latent space through learnable linear transformations followed by layer normalization to reduce distribution discrepancies.

  1. Bold header: Fine-Grained Subtype Classification

The improved system can perform fine-grained classification of breast fibroadenoma (FA), benign phyllodes tumor (B), and borderline phyllodes tumor (BD) with high accuracy, maintaining strong performance even when class distribution is balanced or simplified to a single PT category.

Sources

Related papers