Longitudinal NSCLC Treatment Progression via Multimodal Generative Models

arXiv:2603.06147 · cs.CV · Submitted 2026-03-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Longitudinal NSCLC Treatment Progression via Multimodal Generative Models".

Jane: Here's the gist of the paper. The gist: This work introduces a Virtual Treatment (VT) framework that formulates non-small cell lung cancer (NSCLC) progression as a dose-aware multimodal conditional image-to-image translation problem, demonstrating that diffusion-based models benefit more consistently from multimodal, dose-aware conditioning and produce more stable and anatomically plausible tumor evolution trajectories than GAN-based baselines Virtual Treatment (VT) Framework The VT framework is introduced to formulate longitudinal NSCLC tumor evolution as a dose-aware multimodal conditional image-to-image translation problem.

Tom: Given a CT scan, baseline clinical variables, and a specified radiation dose increment, VT aims to synthesize plausible follow-up CT images reflecting treatment-induced anatomical changes.

Jane: The main contributions of this work include introducing the "first Virtual Treatment framework for radiotherapy that explicitly formulates NSCLC evolution prediction as a dose-aware multimodal conditional image-to-image translation problem".

Tom: Furthermore, the authors propose a "tumor-focused loss that restricts the reconstruction objective to the Clinical Target Volume (CTV), using the baseline planning segmentation as an anatomical supervision that guides the models toward clinically relevant regions".

Title and authors: Tom: The abstract says they use a private longitudinal dataset of two hundred twenty-two patients with stage III NSCLC treated with radiotherapy at Fondazione Policlinico Universitario Campus Bio-Medico di Roma.

Jane: They have about eight hundred ninety-five thoracic CT scans in total, including one baseline simulation scan and several follow-up scans for each patient.

Lu: For the conditioning data, they collect age, sex, histological diagnosis, and clinical tumor and nodal staging variables at the time of treatment planning as their baseline clinical variables.

Meng: And then every single acquisition is paired with a cumulative delivered dose recorded at that time point, which is used as the only longitudinal clinical variable for tracking changes.

Tom: The multimodal conditioning vector they use, denoted as h delta, is defined by concatenating those baseline clinical variables with the specific dose increment being considered.

Jane: So they are feeding this combination of patient history and the dose change into their generative model to predict future scans.

Lu: The generative formulation aims to learn that conditional distribution p theta x p s given an observation x p t, cp, and a dose increment delta p t to s.

Tom: And they define the training set Dtrain as all those ordered scan pairs where the first time point is earlier than the second time point for every patient in their training cohort.

Jane: That construction ensures that for each patient, they are trained on every possible sequence of admissible longitudinal transitions, which is a really thorough way to build up the learning experience.

Meng: It sounds comprehensive for training data, but I wonder how much real-world noise is filtered out when you use this specific dataset structure.

Tom: That’s a valid concern. But the paper seems confident that this setup captures the essential dynamics of treatment-induced anatomical changes in NSCLC.

The paper's summary: Jane: They focus on two main contributions, and the first is introducing this Virtual Treatment framework for radiotherapy that explicitly formulates NSCLC evolution prediction as a dose-aware multimodal conditional image-to-image translation problem.

Tom: So the improvement isn't just in predicting the scan; it’s in *how* they frame the entire process—making it dose-aware and multimodal from the start.

Lu: The second major contribution is proposing this tumor-focused loss that restricts the reconstruction objective to just the Clinical Target Volume or CTV.

Jane: That means they are using their baseline planning segmentation as an anatomical supervision to guide the models toward those clinically relevant regions, which is a big deal for clinical applicability.

Meng: That’s what I mean by guiding the model; it prevents it from learning irrelevant artifacts in healthy tissue while focusing its generation effort on where the tumor actually is.

Tom: And they show that this approach leads to a better result when comparing GAN-based and diffusion-based generative models across 2D and 2 point 5D formulations, analyzing both volumetric tumor evolution and computational efficiency under dose-aware multimodal conditioning <ref:2603.06147#pg1>.

Jane: That comparison shows that the diffusion-based model, TADM, produces more stable trajectories and moderate variation at higher dose levels compared to the GANs they tested.

Lu: The paper suggests that this diffusion-based approach benefits more consistently from multimodal, dose-aware conditioning and produces more stable and anatomically plausible tumor evolution trajectories than GAN-based baselines.

Tom: That’s a big win for stability, especially when dealing with higher dose increments where the differences between models usually start to get worse.

The paper's improvements: Jane: To wrap up, what does this whole study mean for someone who is just listening to this show? It suggests that AI models can be much more robust when we give them context about the dose level and clinical variables.

Tom: It points toward in-silico treatment monitoring and adaptive radiotherapy support because these diffusion-based models, specifically TADM, gave the best balance between predictive stability and computational cost.

Lu: The paper suggests that for this specific problem, diffusion-based models demonstrated greater robustness and produced more stable volumetric predictions across dose increments than the GANs they compared against.

Meng: From a deployment perspective, that computational efficiency is crucial; TADM showed a substantial reduction in operations when moving from training to inference compared to the other models.

Lalam: I am very sure this finding holds up because the results show TADM provided the best balance between predictive stability and computational cost, which is exactly what you need for real-world use cases.

Tom: So, in summary, by introducing this Virtual Treatment framework and focusing on dose-aware conditioning and tumor-focused loss, they showed that diffusion models are more reliable for these longitudinal predictions than GANs when dealing with dose increments.

Jane: That’s the main point—we are moving toward a system that can generate synthetic treatment trajectories that respect the clinical constraints of dose delivery.

Lu: It opens up possibilities for exploring alternative dose-response scenarios in silico without needing actual patient follow-up data to see how the tumor might react.

Meng: And it gives us a concrete direction on which model architecture is likely to be more practical for deploying these types of forecasting tools in a clinical setting.

Conclusion: Tom: So we've been talking about how this Virtual Treatment framework uses dose increments and clinical variables to predict NSCLC tumor evolution, and now we’re getting to the finish line of this paper, "Longitudinal NSCLC Treatment Progression via Multimodal Generative Models."

Jane: Right, they compared a diffusion model against GANs and found that the diffusion model—TADM—gave more stable trajectories and better results at higher dose levels.

Lu: It’s interesting that they used a tumor-focused loss restricted to the Clinical Target Volume, which acts as a strong anatomical guide during the image-to-image translation.

Meng: From an engineering standpoint, TADM had a much better trade-off between how accurate it was and how much compute power it needed to run.

Lalam: I'm confident that this finding about diffusion models being more robust under multimodal conditioning is solid because the stability they showed across different dose increments really stands out.

Tom: It means for adaptive radiotherapy, we might be better off leaning on these diffusion-based forecasting methods when we need reliable predictions under varying treatment doses.

Jane: Exactly, it suggests a more reliable way to do in-silico monitoring before actually starting the treatment plan.

Lu: The ability to query the model at arbitrary dose levels also means you can explore alternative scenarios without waiting for real patient follow-up data, which is a big concept for planning.

Meng: It’s efficient because it doesn't require that massive computational overhead that some of these other generative models have when you try to run them in production.

Lalam: So we are looking at a system where the conditioning vector h delta—the mix of clinical data and dose increment—is key to getting those accurate, stable predictions.

Tom: yeah, it really shows how important that multimodal input is for these types of image translation tasks.

Jane: It’s a solid foundation for future work on developing more adaptive treatment support tools in oncology.

Lu: I think the next step would be to explore how these models handle even more complex, time-dependent biological factors beyond just dose and age.

Meng: I wonder what kind of real-world data we need to make this framework truly robust when we move it from simulation into actual clinical settings.

Lalam: We'll see what those next steps look like as the field continues to explore these multimodal generative approaches for medical imaging.

Massimiliano Mantegna, Elena Mulero Ayllón, Alice Natalina Caragliano, Francesco Di Feola, Claudia Tacconi, Michele Fiore, Edy Ippolito, Carlo Greco, Ūara Philippe C. Cattin, Paolo Soda, Matteo Tortora, Valerio Guarrasi

Unit of Artificial Intelligence and Computer Systems at Universita Campus Bio-Medico di Roma · Multi-Specialist Clinical Institute for Orthopaedic Trauma Care at Messina · Department of Diagnostics and Intervention at UmeÔ University in Sweden · Operative Research Unit of Radiation Oncology at Fondazione Policlinico Universitario Campus Bio-Medico di Roma · Department of Biomedical Engineering at University of Basel, Allschwil, Switzerland · Department of Naval, Electrical, Electronics and Telecommunications Engineering at University of Genoa

cs.CV

Submitted: 2026-03-06

Updated: 2026-03-06

DOI: 10.1109/CBMS69103.2026.00130

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 83/100

The gist: The gist: This work introduces a Virtual Treatment (VT) framework that formulates non-small cell lung cancer (NSCLC) progression as a dose-aware multimodal conditional image-to-image translation

Key concepts

Virtual Treatment (VT) Framework
This framework treats tumor growth over time as a conditional image translation task. It takes an initial scan, clinical data, and a dose increase to generate a plausible future scan. This allows clinicians to simulate how the tumor might change based on specific radiation doses received.
Dose-Aware Multimodal Conditioning
The model uses two types of input: baseline patient information (like age and staging) and the specific increment in radiation dose. By combining these, the system learns to predict anatomical changes that are directly related to the dose administered, leading to more accurate predictions.
Tumor-Focused Loss
This is a special part of the training loss function that forces the model to focus its reconstruction efforts specifically on the Clinical Target Volume (CTV). This ensures that when synthesizing new images, the model prioritizes accurately depicting the area where treatment is intended to affect.
Diffusion Models vs. GANs
The study compared diffusion models (TADM) against Generative Adversarial Networks (GANs). Diffusion models were superior because they produced more stable and anatomically plausible tumor evolution predictions, especially at higher dose levels, avoiding the overestimation of tumor shrinkage seen in GAN-based methods.

Terminology

Summary

The gist: This work introduces a Virtual Treatment (VT) framework that formulates non-small cell lung cancer (NSCLC) progression as a dose-aware multimodal conditional image-to-image translation problem, demonstrating that diffusion-based models benefit more consistently from multimodal, dose-aware conditioning and produce more stable and anatomically plausible tumor evolution trajectories than GAN-based baselines

Virtual Treatment (VT) Framework

The VT framework is introduced to formulate longitudinal NSCLC tumor evolution as a dose-aware multimodal conditional image-to-image translation problem. Given a CT scan, baseline clinical variables, and a specified radiation dose increment, VT aims to synthesize plausible follow-up CT images reflecting treatment-induced anatomical changes. The main contributions of this work include introducing the first Virtual Treatment framework for radiotherapy that explicitly formulates NSCLC evolution prediction as a dose-aware multimodal conditional image-to-image translation problem. Furthermore, the authors propose a "tumor-focused loss that restricts the reconstruction objective to the Clinical Target Volume (CTV), using the baseline planning segmentation as an anatomical supervision that guides the models toward clinically relevant regions".

Training Data and Conditioning

The study uses a private longitudinal dataset of 222 patients with stage III NSCLC treated with radiotherapy at Fondazione Policlinico Universitario Campus Bio-Medico di Roma. This cohort comprises 895 thoracic CT scans, including one baseline simulation CT per patient and multiple follow-up CT scans acquired during treatment. For each acquisition, the cumulative delivered dose was recorded and used as the only longitudinal clinical variable. Baseline variables collected at treatment planning include age, sex, histological diagnosis, and clinical tumor and nodal staging (cT, cN). The multimodal conditioning vector hδ is defined as concatenating baseline clinical variables and dose increment.

Generative Formulation and Loss Function

The goal of VT is to learn the conditional distribution pθ xp,s given an observation (xp,t, cp) and a dose increment δp,t→s. This is modeled by an image-to-image generator Gθ producing a prediction xˆp,s of the future scan xp,s. The training set Dtrain is defined as all ordered scan pairs (t, s) with t < s for every patient in Ptrain. A composite loss function Ltotal is employed, combining the standard objective of the generative model with a tumor-focused term: Ltotal = Lmain + λLtumor. The tumor-focused loss encourages accurate reconstruction of the clinically most relevant region, defined by M(i)p,t = 1 if voxel i belongs to the CTV and 0 otherwise.

Model Comparison and Results

The evaluation benchmarks two model families: (i) 2D GAN-based architectures (Pix2Pix and CycleGAN) and (ii) a 2.5D diffusion-based model (TADM), which is adapted to model treatment-driven, rather than age-driven, anatomical progression. The volumetric tumor evolution is assessed using the relative volumetric discrepancy ∆V = 100 · Vpred − Vreal / Vreal. Results indicate that TADM produces more stable trajectories and moderate variation at higher dose levels, whereas CycleGAN and Pix2Pix show a progressive overestimation of tumor shrinkage that becomes more pronounced for large dose increments.

Computational Efficiency

A clear accuracy-complexity trade-off emerges in the computational cost analysis. Diffusion-based TADM achieves substantially improved accuracy at moderate computational cost, and the gap between training and inference cost differs markedly across models. Specifically, when transitioning from training to inference, the number of required operations is reduced by approximately 53% for CycleGAN, 8% for Pix2Pix, and 78% for TADM. This highlights a substantially more favorable deployment profile for diffusion-based forecasting in longitudinal settings.

Conclusion

Diffusion-based models demonstrated greater robustness and produced more stable volumetric predictions across dose increments, whereas GAN-based models exhibited progressive error accumulation and overestimation of tumor shrinkage at higher doses. TADM provided the best balance between predictive stability and computational cost. This framework represents a promising step toward in-silico treatment monitoring and adaptive radiotherapy support in NSCLC.

Acknowledgments

Massimiliano Mantegna and Alice Natalina Caragliano are PhD students enrolled in the National PhD program in Artificial Intelligence, in the XXXVIII and XXXIX cycle, respectively, course on Health and life sciences, organized by Università Campus Bio-Medico di Roma. They also acknowledge partial financial support from Università Campus BioMedico di Roma within the project “AI-powered Digital Twin for next-generation lung cancer care (IDEA)”, JCSMK24-0094, and Cancerforskningsfonden Norrland project LP 25-2383. They also provide resources by NAISS and SNIC at Alvis @ C3SE (grants no. 2022-06725 and 2018-05973).

References

[1] N. J. Schork, Artificial intelligence and personalized medicine, in: Precision medicine in Cancer therapy, Springer, 2019, pp. 265–283

[2] M. Tortora, et al., RadioPathomics: multimodal learning in non-small cell lung cancer for adaptive radiotherapy, IEEE Access 11 (2023) 47563–47578

[3] C. M. Caruso, et al., A deep learning approach for overall survival prediction in lung cancer with missing values, Computer Methods and Programs in Biomedicine 254 (2024) 108308

[4] D. Paolo, et al., Predicting lung cancer survival with attention-based CT slices combination, Health Information Science and Systems 14 (1) (2026) 20

[5] L. Nibid, et al., Deep pathomics: A new image-based tool for predicting response to treatment in stage iii non-small cell lung cancer, Plos one 18 (11) (2023) e0294259

[6] A. N. Caragliano, et al., Doctor-in-the-loop: An explainable, multi-view deep learning framework for predicting pathological response in nonsmall cell lung cancer, Image and Vision Computing 161 (2025) 105630

[7] C. Z. Liu, et al., Exploring deep pathomics in lung cancer, in: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), IEEE, 2021, pp. 407–412

[8] A. N. Caragliano, et al., Multimodal doctor-in-the-loop: A clinicallyguided explainable framework for predicting pathological response in non-small cell lung cancer, in: 2025 International Joint Conference on Neural Networks (IJCNN), 2025, pp. 1–8

[9] V. Guarrasi, et al., A systematic review of intermediate fusion in multimodal deep learning for biomedical applications, Image and Vision Computing (2025) 105509

[10] M. Baumann, et al.

Improvements for AI systems

  1. Bold header: Dose-Aware Multimodal Conditioning Integration

The VT framework explicitly formulates NSCLC progression as a dose-aware multimodal conditional image-to-image translation problem by conditioning generation on delivered dose increments together with demographic and tumor-related clinical variables. This allows the improved system to synthesize plausible follow-up CT images reflecting treatment-induced anatomical changes that are directly linked to the specific radiation dose received.

  1. Bold header: Tumor-Focused Reconstruction Objective

The system incorporates a specific regularization term, Ltumor encourages accurate reconstruction of the clinically most relevant region, defined as a loss restricted to the Clinical Target Volume (CTV), ensuring that synthesis prioritizes clinically relevant regions guided by baseline segmentation.

  1. Bold header: Diffusion Model Superiority for Stability

By benchmarking diffusion-based models against GANs, the system leverages the finding that diffusion-based models benefit more consistently from multimodal, dose-aware conditioning and produce more stable and anatomically plausible tumor evolution trajectories than GAN-based baselines. This makes the improved system superior for longitudinal forecasting by producing predictions with moderate variation at higher dose levels compared to GANs' progressive overestimation of tumor shrinkage.

  1. Bold header: Adaptive Dose-Response Inference

The inference strategy enables a powerful capability: The model can be queried at arbitrary dose levels, including hypothetical values, thus generating a synthetic dose-indexed progression trajectory decoupled from the clinical acquisition schedule. This allows for exploring alternative dose-response scenarios in silico without waiting for actual patient follow-up.

  1. Bold header: Computational Efficiency Optimization

The system demonstrates a favorable trade-off by showing that TADM providing the best balance between predictive stability and computational cost, particularly noting the significant reduction in operations when moving from training to inference, which is crucial for large-scale virtual inference in adaptive radiotherapy.

Abstract

Predicting tumor evolution during radiotherapy is a clinically critical challenge, particularly when longitudinal changes are driven by both anatomy and treatment. In this work, we introduce a Virtual Treatment (VT) framework that formulates non-small cell lung cancer (NSCLC) progression as a dose-aware multimodal conditional image-to-image translation problem. Given a CT scan, baseline clinical variables, and a specified radiation dose increment, VT aims to synthesize plausible follow-up CT images reflecting treatment-induced anatomical changes. We evaluate the proposed framework on a longitudinal dataset of 222 stage III NSCLC patients, comprising 895 CT scans acquired during radiotherapy under irregular clinical schedules. The generative process is conditioned on delivered dose increments together with demographic and tumor-related clinical variables. Representative GAN-based and diffusion-based models are benchmarked across 2D and 2.5D configurations. Quantitative and qualitative results indicate that diffusion-based models benefit more consistently from multimodal, dose-aware conditioning and produce more stable and anatomically plausible tumor evolution trajectories than GAN-based baselines, supporting the potential of VT as a tool for in-silico treatment monitoring and adaptive radiotherapy research in NSCLC.

Sources

Related papers