GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image

arXiv:2412.06129 · cs.CV · Submitted 2024-12-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image".

Jane: Here's the gist of the paper. The gist The GCUNet network, a GNN-based contextual learning network, was proposed to enhance tertiary lymphoid structure semantic segmentation in whole slide images by capturing long-range and fine-grained contextual information outside the target patch.

Tom: Tertiary lymphoid structure (TLS) is an aggregate of immune cells that can be classified into three levels of maturity: early TLS (E-TLS), primary follicular-like TLS (PETTLS) and secondary follicle-like TLS (SEL-TLS)

2, 31: .

Jane: In most solid tumors, the presence of TLS is closely associated with the anti-tumor immune response, which is significantly influenced by the maturity of TLS

2: .

Tom: To identify the maturity levels of TLS, multiplex-immunohistochemistry (mIHC) is commonly used to detect specific molecular expressions such as CD21+ for PET-TLS and CD23+ for SEL-TLS

3: .

Paper summary: Tom: We've just heard that GCUNet uses a Graph Neural Network to look at spatial relationships between patches for segmentation. The paper claims they are moving beyond simple boundaries to identify the actual maturity stages of these structures.

Jane: That’s right, and they use this GNN structure because it lets them aggregate context from outside the target patch, which is something traditional patch-based strategies can't do well <ref:2412.06129#pg2>.

Lu: They build a context graph where patches are nodes and edges connect them based on spatial connectivity, allowing for both long-range and fine-grained information gathering <ref:2412.06129#pg2>.

Meng: It seems like they’re tackling the problem of missing outside information, which is a big hurdle in whole slide image analysis <ref:2412.06129#pg3>.

Tom: And they do have a fusion block called DCFusion to integrate that gathered context with the details of the target patch to make the final segmentation mask <ref:2412.06129#pg1>.

Jane: So, it’s about taking those contextual features and combining them with the detailed features of the patch at a semantic level to predict where things are <ref:2412.06129#pg1>.

Lu: The network progressively aggregates this information over T0 aggregation steps so that each node gets updated with context from increasingly distant neighbors <ref:2412.06129#pg2>.

Tom: And they tested this on four datasets: TCGA-COAD, TCGA-LUSC, TCGA-BLCA and INHOUSE-PAAD <ref:2412.06129#pg5>.

Jane: The main point here is that the success of GCUNet hinges on its ability to capture information outside the patch because that's what allows it to distinguish between the different TLS maturity stages <ref:2412.06129#pg8>.

Conclusion: Tom: So we’ve talked about "GCUNet: A GNN-Based Contextual Learning Network for Tertiary Lymphoid Structure Semantic Segmentation in Whole Slide Image" and how it works. This whole approach is designed to move past just drawing lines on an image to actually understanding the different stages of immune cell aggregation in real tissue.

Jane: Right, so we started by asking what this paper actually does and then we dug into how that GNN structure helps it capture the maturity levels better than before <ref:2412.06129#pg1>.

Lu: What’s really interesting is how they use a graph to connect patches together spatially, which means it’s not just looking at one spot in isolation anymore <ref:2412.06129#pg2>.

Meng: I wonder what the actual training cost is for this kind of GNN on these massive whole slide images when you try to scale it up for real clinical use <ref:2412.06129#pg1>.

Lalam: Based on the literature, the key is that this contextual information outside the target patch is what allows it to distinguish between those three distinct maturity levels <ref:2412.06129#pg8>.

Jane: So if we take what we just talked about—this GNN looking around—what does that actually mean for a clinician trying to diagnose a tumor?

Tom: It means they can get a more accurate picture of the immune response by knowing not just what's in the patch, but how it relates to its neighbors <ref:2412.06129#pg8>.

Lu: The ablation studies showed that increasing the number of GCN layers helps propagate that long-range information, and having three layers was actually where they hit their best performance <ref:2412.06129#pg1>.

Meng: From a deployment standpoint, if we’re talking about putting this into a pipeline, the complexity of the DCFusion block is something I’d want to keep an eye on for latency <ref:2412.06129#pg5>.

Jane: It sounds like the main idea is that context beyond the immediate area is what unlocks that better segmentation of TLS maturity >

Lei Su, Zonghao Liu, Junxian Wu, Zewen Sun, Jiaxuan Wen, Lu Zhu, Xuqing Geng, Tianwang Xun, Jing Kuang, Lizhi Shao, Yang Du

CASMI, Institute of Automation, Chinese Academy of Sciences (CASIA) · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital · School of Computer Science and Engineering, Southeast University · Institute of Pathology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology · School of Internet, Anhui University

cs.CV

Submitted: 2024-12-09

Updated: 2024-12-09

Journal ref: Visual Computing for Industry, Biomedicine, and Art 9, 15 (2026)

DOI: 10.1186/s42492-026-00227-z

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist The GCUNet network, a GNN-based contextual learning network, was proposed to enhance tertiary lymphoid structure semantic segmentation in whole slide images by capturing long-range and

Key concepts

Tertiary Lymphoid Structure (TLS): TLS are aggregates of immune cells found in solid tumors. They exist in three distinct maturity levels: early TLS (E-TLS), primary follicular-like TLS (PETTLS), and secondary follicle-like TLS (SEL-TLS). Identifying these levels is crucial for understanding the anti-tumor immune response, which is often linked to tumor prognosis.
Patch-based Strategy Limitation: Because Whole Slide Images are extremely large, segmentation is usually done by analyzing small patches. This limits the model's view to only what's inside the patch, preventing it from accessing important contextual information located outside that specific area. GCUNet overcomes this by looking beyond the patch.
Graph Neural Network (GNN): A GNN is a type of neural network that models data as a graph, where patches are nodes and spatial connections are edges. This structure allows the model to iteratively aggregate information from neighboring patches, enabling it to capture both local details and broader, long-range contextual relationships across the entire image.
Detail and Context Fusion (DCFusion): DCFusion is a block designed to combine two types of input: detailed features from the target patch and contextual features aggregated from the surrounding area. By fusing these two information streams at a semantic level, DCFusion allows GCUNet to make more accurate predictions about the boundaries and maturity levels of TLS.

Terminology

Semantic Segmentation — 6× in this paper · explained in 3 episodes

Summary

The gist The GCUNet network, a GNN-based contextual learning network, was proposed to enhance tertiary lymphoid structure semantic segmentation in whole slide images by capturing long-range and fine-grained contextual information outside the target patch.

Introduction and Motivation

Tertiary lymphoid structure (TLS) is an aggregate of immune cells that can be classified into three levels of maturity: early TLS (E-TLS), primary follicular-like TLS (PETTLS) and secondary follicle-like TLS (SEL-TLS) [2, 31]. In most solid tumors, the presence of TLS is closely associated with the anti-tumor immune response, which is significantly influenced by the maturity of TLS [2]. To identify the maturity levels of TLS, multiplex-immunohistochemistry (mIHC) is commonly used to detect specific molecular expressions such as CD21+ for PET-TLS and CD23+ for SEL-TLS [3]. The identification of TLS in WSI is important in tumor diagnosis and treatment [3]. Binary segmentation is a critical approach for delineating the boundaries of TLS [19, 33, 34, 39, 40]

Proposed Method: GCUNet Architecture

The proposed GCUNet architecture consists of two main steps: first, the multi-layer GCN iteratively aggregates long-range and fine-grained contextual information outside target; then, DCFusion integrates the context and detail of target at semantic level to predict segmentation mask

The context graph G = (V, E) is constructed where V denotes the set of patch features, and E represents the set of edges that connect patches based on spatial connectivity. Each patch pi is encoded into a feature vector vi ∈ R 1024 using UNI, a transformer-based vision encoder pre-trained on millions of pathology images. The aggregation of contextual information outside the patches over t steps can be expressed as X(t) = FGCN(X(t−1)) = σ(AX˜ (t−1)W(t−1)). After T0 aggregation steps, the feature of node i is updated from x(0)i to x(T0)i, incorporating contextual information from increasingly distant neighbors. This process enables the feature of the node to capture long-range contextual information.

The Detail and Context Fusion block (DCFusion) is designed to perform a semantic-level fusion of the contextual and detailed information. The fused feature z(0)i = [z c i +epos; z d i] ∈ R(b 2+1)×L, where z c i is the long-range contextual features and z d i is the detailed features. DCFusion consists of l layers of multi-head attention (MSA) and a fully connected block to compute the final fused feature.

Experimental Setup and Results

The model was evaluated on four TLS semantic segmentation datasets, including TCGA-COAD, TCGA-LUSC, TCGA-BLCA and INHOUSE-PAAD. Experiments on these datasets demonstrate the superiority of GCUNet, achieving at least 7.41% improvement in mF1 compared with SOTA.

The performance comparison shows that GCUNet significantly outperforms other methods across all four evaluation metrics. On the TCGA-COAD dataset, GCUNet also outperforms the second-best model HookNet, with improvements of 11.39% in mF1 and 12.96% in mIOU. Furthermore, GCUNet achieves the best performance on the TCGA-LUSC dataset, where mF1 and mIOU are improved by 0.158 and 0.105 over the baseline.

Ablation Studies

Ablation studies investigated several factors influencing model performance. The number of GCN layers influences how far contextual information can be propagated for the target patch, and the model achieves the best results when the number of GCN layers is set to 3. Information granularity suggests that at a spatial resolution of 1 µm/pixel, the model achieved relatively balanced performance across both background and TLS categories. Finally, DCFusion achieved the best performance on both datasets, achieving mF1 increased by 9.25% and 17.3% over baseline.

The paper concludes that the improvement in model performance is primarily due to the ability to distinguish the three maturity levels and the long-range, fine-grained contextual information provided by GNNs. The results highlight that contextual information outside the target patches is crucial for TLS semantic segmentation. The superior segmentation of the three TLS maturity by GCUNet confirms the effectiveness of the proposed model.

--- Page 8 ---

Figure 3 show that GCUNet achieves the most accurate segmentation, closely matching the ground truth labels, particularly in capturing fine-grained details and distinguishing between TLS maturity levels. The superior segmentation of the three TLS maturity by GCUNet confirms the effectiveness of the proposed model.

--- Page 9 ---

In this paper, we introduced a new task of TLS semantic segmentation in WSI and proposed a GNN-based contextual learning network GCUNet. GCUNet used GCNs to flexibly aggregate long-range and fine-grained contextual information beyond the target patch, while the designed DCFusion performed semantic-level fusion of detailed and contextual information to predict patch masks. We collected four TLS semantic segmentation datasets and released annotations for three of them (TCGA-COAD, TCGA-LUSC, and TCGA-BLCA), comprising 826 WSIs and 15,276 TLSs. Our results on these datasets demonstrated the superiority of GCUNet.

--- Page 5 ---

Table 5 demonstrate that all fusion strategies effectively utilize contextual information. Among the fusion strategies, Cat outperformed other basic strategies. DCFusion achieved the best performance on both datasets, achieving mF1 increased by 9.25% and 17.3% over baseline.

--- Page 7 ---

Table 4 shows that GCUNet achieves the most accurate segmentation, closely matching the ground truth labels, particularly in capturing fine-grained details and distinguishing between TLS maturity levels. The superior segmentation of the three TLS maturity by GCUNet confirms the effectiveness of the proposed model.

--- Page 8 ---

The performance of TLS semantic segmentation is hindered by contextual information outside the target. The improvement in model performance is primarily due to the ability to distinguish the three maturity levels and the long-range, fine-grained contextual information provided by GNNs.

--- Page 5 ---

We focus on a new task, i.e., TLS semantic segmentation in WSI. To the best of our knowledge, we are the first to capture contextual information outside of the target patch for TLS semantic segmentation.

--- Page 3 ---

We build four TLS semantic segmentation datasets, called TCGA-COAD, TCGA-LUSC, TCGA-BLCA and INHOUSE-PAAD, and make the former three datasets (comprising 826 WSIs and 15,276 TLSs) publicly available to promote the TLS semantic segmentation.

--- Page 3 ---

The main contributions of our method include: • We focus on a new task, i.e., TLS semantic segmentation in WSI.

Improvements for AI systems

Bold Header: Contextual Learning for Patch-Level Segmentation

GCN layers are used to progressively aggregate long-range and fine-grained contextual information outside target patches, addressing the limitation of patch-based strategies that prevents the model from accessing information outside of the patches. This allows the system to discover discriminative features by leveraging context beyond the immediate target region.

Bold Header: Semantic Fusion for Enhanced Prediction

The Detail and Context Fusion block (DCFusion) is designed to perform a semantic-level fusion of contextual and detailed information, combining features such that the distinctiveness of the overall context in the detailed features z d i is enhanced by the addition of positional encoding. This fusion leads to a predicted segmentation mask with superior performance, as shown by the observation that DCFusion achieved the best performance on both datasets.

Bold Header: Multi-Stage TLS Classification

The model can perform TLS semantic segmentation across three maturity stages (E-TLS, PET-TLS, and SEL-TLS) by leveraging both detailed and contextual data. This capability enables the system to precisely delineate boundaries and assess the maturity levels of tertiary lymphoid structures in whole slide images (WSI).

Bold Header: Robustness through Contextual Depth

The performance improvement is attributed to the ability to distinguish the three maturity levels and the long-range, fine-grained contextual information provided by GNNs. This suggests that deep aggregation steps, specifically setting the number of GCN layers to 3, yields the best results for context propagation.

Bold Header: Fine-Grained Feature Sensitivity

The system's segmentation is sensitive to input resolution and information granularity; specifically, As the spatial resolution increases, the model becomes better at distinguishing the background, but the performance on TLS decreases significantly. This indicates that maintaining a pixel spatial resolution of 1 µm/pixel is optimal for balancing performance across both background and TLS categories.

Bold Header: Fusion Strategy Optimization

The DCFusion strategy is superior to basic methods like concatenation or dot product, as it utilizes a self-attention mechanism for semantic-level fusion of the two types of information, achieving a mF1 increased by 9.25% and 17.3% over baseline on the respective datasets.