Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports

summary

Video file (mp4)

The gist

The gist The present feasibility study aims to demonstrate the use of open-source LLMs in clinical settings for secure and efficient analysis of text-based CMR imaging reports.

In short

The study tested open-source Large Language Models (LLMs) locally within hospitals to analyze text from cardiac MRI reports securely, avoiding data sharing risks. Results show these models perform very well, with some achieving F1 scores above 0.93. The research proves that open-source LLMs can effectively extract diagnostic categories from complex CMR reports while offering a path to privacy-preserving clinical analysis.

Key concepts

CMR Imaging Reports
These are detailed written summaries generated from cardiovascular magnetic resonance (CMR) scans. They contain both structured quantitative data and unstructured narrative text describing findings related to heart conditions like hypertrophic cardiomyopathy or myocarditis, which the LLMs analyze.
Open-Source LLMs Locally Deployed
This involves running Large Language Models on a hospital's own secure network instead of using external cloud services. This approach keeps sensitive patient data private and compliant with regulations like GDPR by processing information internally.
F1 Score
The F1 score is a metric used to evaluate how accurately an LLM classifies reports into different cardiac diagnostic categories. A high F1 score, such as the 0.98 achieved by Gemma 2, indicates that the model is highly effective at correctly identifying the correct diagnosis.
Computational Efficiency vs. Performance
This concept explores the trade-off between how fast a model runs and how accurate its results are. The study found that while very large models like DeepSeek-R1 (70B) are slow, they don't always provide better accuracy than smaller models, suggesting efficiency is crucial for clinical use.

Terminology used across episodes

This episode discusses

The paper

Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports · Read on arXiv

D-Lab, Department of Precision Medicine, GROW - Research Institute for Oncology and Reproduction, Maastricht University · Division of Cardiovascular Imaging, Department of Cardiology I, University Hospital Munster

Purpose: We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. Materials and Methods: We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Results: Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google's Gemma2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32B with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral and DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Conclusion: Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.

DOI: 10.1038/s41598-026-75150-7

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports".

Jane: The gist The present feasibility study aims to demonstrate the use of open-source LLMs in clinical settings for secure and efficient analysis of text-based CMR imaging reports.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports. It's about taking those complex text reports and seeing how well these open-source models can handle them securely.

Jane: Exactly, Tom, it tackles the big problem of using cloud services for patient data when you need to keep things private under regulations like GDPR. The authors are showing that you can run the LLM right there in the hospital network.

Lu: This paper is focused on demonstrating how to use locally deployed open-source Large Language Models to extract diagnostic information from free-text cardiovascular magnetic resonance reports, which is a direct response to those privacy concerns.

Meng: So, what they're doing here is testing nine different open-source LLMs against one hundred nine clinical CMR reports from places like University Hospital Munster in Germany to see who can identify diagnoses and classify patients into various cardiac categories <ref:2506.00060#pg1,identify diagnoses and classify patients into various cardiac>.

Tom: The core idea is that this setup allows for a secure analysis of those text-based CMR reports without the data ever leaving the hospital's secure network.

Jane: It’s about making sure that this approach isn't just theoretical, but shows how it works in a real clinical setting with actual patient reports.

The paper's summary: Tom: The paper summarizes their findings by showing that most open-source LLMs achieved excellent accuracy and precision across all the tests, generally above a score of zero point nine three <ref:2506.00060#pg2>.

Jane: That’s a really high bar, Tom, because when you look at the results, the Gemma-two-27b model demonstrated the highest overall performance with an average score exceeding zero point nine seven across all metrics <ref:2506.00060#pg2>.

Lu: The confusion matrices they showed are important because they highlight where the models struggle, and they found strong diagonal dominance in models like Gemma2-27b, DeepSeek-R1:32b, and Qwen2-five:32b <ref:2506.00060#pg3>.

Meng: But even with those high scores, there were some specific challenges where the models mixed up similar diseases, like myocarditis being sometimes misclassified as ischemic or dilated cardiomyopathy <ref:2506.00060#pg3>.

Jane: That’s a critical detail because in medicine, confusing similar diagnoses can have serious consequences for patient care, so that's something the authors highlight <ref:2506.01#pg3>.

Tom: They also looked at the speed of these models and showed that smaller models like LLaMA three with two point three billion parameters had an average execution time of zero point four one seconds per patient <ref:2506.01#pg3>.

Lu: But they pointed out a trade-off; larger models, like DeepSeek-R1:70b, had the highest execution time at fifty point seven seven seconds per patient <ref:2506.06#pg3>.

The paper's improvements: Jane: The paper points toward making these tools practical for actual clinical deployment and showing that local deployment is a viable alternative to cloud solutions that cause privacy worries <ref:2506.01#pg2>.

Tom: Their main improvement is clearly demonstrating that you can use open-source, locally run LLMs as a viable alternative to cloud-based solutions that raise privacy concerns when handling sensitive patient data <ref:2506.01#pg2>.

Lu: They are emphasizing the feasibility study itself, showing that this local deployment method provides a secure way to keep patient data within the hospital’s network and comply with rules like GDPR <ref:2506.01#pg2>.

Meng: From a deployment standpoint, they’re showing that these models can actually operate efficiently enough for high-throughput applications, especially the smaller models that run in under one second per patient <ref:2506.01#pg3>.

Jane: They also touched on what comes next by suggesting analyzing more than four thousand cases using the three best-performing LLMs to generate a consensus result <ref:2506.01#pg3>.

Tom: So the immediate next step they are proposing is to move beyond this small set of one hundred nine reports and test their top models on a much larger dataset to see if those high performance numbers hold up more robustly <ref:2506.01#pg3>.

Conclusion: Tom: So we're wrapping up this look at Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports. Basically, they proved that you can use open-source, locally run LLMs to analyze those text reports securely without sending patient data to the cloud <ref:2506.01#pg2>.

Jane: That’s really interesting because it tackles a huge issue right there—patient privacy. The authors showed that by keeping the models local in the hospital network, you can bypass many of those GDPR concerns <ref:2506.01#pg2>.

Lu: I think what's important is how they structured the pipeline: they split the report into a quantitative part and a narrative text, then used LLaMA three point three to translate that German narrative into English before feeding it to the open-source LLMs like Ollama <ref:2506.01#pg4>.

Meng: From an engineering standpoint, they showed that performance isn't just about picking the biggest model, because a smaller one like LLaMA three with two point three billion parameters was actually much faster at one point, running in zero point four one seconds per patient <ref:2506.06#pg3>.

Lalam: I can confirm what they found: most open-source LLMs achieved classification scores above zero point nine three across the board, with Google’s Gemma two model hitting an average F1 score of zero point nine eight <ref:2506.01#pg6>.

Tom: So they beat the board-certified cardiologist on all metrics, which is a pretty big deal when you're talking about medical interpretation <ref:2506.06#pg3>.

Jane: But they also showed that this isn't perfect; there were still some confusion patterns, like sometimes myocarditis got mislabeled as ischemic or dilated cardiomyopathy <ref:2506.01#pg3>.

Lu: That misclassification detail is crucial because it highlights the specific diagnostic challenges in cardiology that even these top models struggle with occasionally <ref:2506.01#pg3>.

Meng: And you can't ignore the efficiency trade-off; the largest model, DeepSeek-R1:70b took fifty point seven seven seconds per patient, which is way too slow for real-time clinical use <ref:2506.06#pg3>.

Tom: It’s a clear message then: you have to balance that computation complexity against the speed required for actual deployment in a clinic <ref:2506.01#pg3>.

Jane: So, the paper confirms that using open-source LLMs locally is feasible for secure analysis of CMR reports, offering a way to automate report processing and support clinical decisions <ref:2506.01#pg3>.

Lu: Their next step is to take this feasibility and scale it up by analyzing more than four thousand cases with the three best models to get a consensus result <ref:2506.01#pg3>.

Meng: I'm hoping that next study focuses on how they manage that data pipeline at a larger scale, because running four thousand reports locally is a whole different kind of engineering problem <ref:2506.01#pg3>.

Lalam: I'm confident the consensus will be strong if they stick to those top three performers; I predict a solid performance metric for them on that larger set <ref:2506.01#pg3>.

Tom: Well, that covers what they did with this paper. It shows the mechanics of how we can use local AI for clinical text analysis <ref:2506.01#pg2>.

Jane: It’s a good reminder that when we look at AI in medicine, it's not just about the raw accuracy score, but also about security and real-world speed <ref:2506.01#pg3>.

Lu: Next time we look at models, I want to see how they handle those complex three dee worlds; that’s where the next big challenge lies <ref:2506.01#pg4>.

More episodes

← Home