Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports

The gist The present feasibility study aims to demonstrate the use of open-source LLMs in clinical settings for secure and efficient analysis of text-based CMR imaging reports.

Introduction and Motivation

Cardiovascular magnetic resonance (CMR) imaging offers unique insights into myocardial viability, inflammation, infiltration and fibrosis1. CMR interpretation requires synthesizing complex imaging data into focused and clinically meaningful reports which often rely on narrative to capture nuanced observations4. The application of LLMs to “clinical report analysis” is associated with substantial challenges concerning patient privacy and data security6. Closed-source LLMs often necessitate accessing their capabilities through APIs, which inherently involves sharing patient data with external servers, raising substantial compliance issues with General Data Protection Regulation (GDPR)10. An alternative approach involves the deployment of locally run, open-source LLMs within hospital infrastructures to keep patient data within the hospital’s secure network and facilitate compliance with stringent regulations like GDPR11.

Methodology and Preprocessing

The study retrospectively analyzed 109 CMR imaging reports from patients scanned at the Cardiac MRI Center of the University Hospital Munster in Germany, including various cardiac diseases such as hypertrophic cardiomyopathy (HCM), cardiac amyloidosis (CA), and myocarditis (MYO)2. Each CMR report consists of two sections: the first provides structured quantitative data, while the second is less structured narrative text describing diagnostic findings4. The preprocessing involved three steps: 1) splitting the report into the first quantitative and second narrative part, 2) translating the narrative text from German to English using a multi-lingual LLaMA3.3 model, and 3) analyzing the English text with open-source LLMs locally using Ollama as an inference framework4.

Model Evaluation and Performance

Nine open-source LLMs were evaluated on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports5. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score6. The results showed that most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories7. Specifically, Google’s Gemma 2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32B with F1 scores of 0.96 and 0.95, respectively8. The top four LLMs outperformed the board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports9.

Classification Patterns and Efficiency

Confusion matrices visualized the relationship between true diagnostic labels and predicted labels, showing strong diagonal dominance particularly in Gemma2-27b, DeepSeek-R1-32B, Qwen2-5-32B, and LLaMA3.310. Normal cardiac findings and HCM were consistently classified with perfect or near-perfect accuracy across these models10. However, some categories showed characteristic misclassification patterns; for example, MYO was occasionally misclassified as ischemic (ICM) or dilated cardiomyopathy (DCM)12. The DeepSeek-R1 model showed the highest rate of misclassification, especially for cardiac amyloidosis (CA), myocarditis, ICM and DCM13. Regarding computational efficiency, smaller models like LLaMA 3 (2.3B) had an average execution time of 0.41 seconds per patient, while larger models such as DeepSeek-R1 (70B) had the highest execution time at 50.77 seconds per patient14. This indicates that increased computation does not necessarily improve performance, underscoring the need to balance model complexity with processing efficiency in clinical settings15.

Conclusion and Future Work

The study demonstrates the feasibility of using open-source, locally run LLMs to analyze CMR reports and derive diagnostic conclusions from text-based clinical data while ensuring privacy16. The findings show that existing LLMs enable classification above 0.93 across most evaluation metrics, highlighting their effectiveness in automating the analysis of imaging reports17. A limitation of this study is the limited number of imaging reports used to evaluate LLM performance and the reliance on single-center data20. As a next step, the researchers aim to analyze more than 4,000 cases using the three best-performing LLMs to generate a consensus result21. The overall results suggest that multiple LLMs can effectively extract diagnostic categories from text-based cardiac imaging reports, offering potential applications in automating imaging report processing and supporting clinical decision-making in cardiology22.

--- Page 1 ---

Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports; Sina Amirrajab PhD1, 2, Volker Vehof MD2, Michael Bietenbeck PhD2, and Ali Yilmaz MD2; The D-Lab, Department of Precision Medicine, GROW - Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands.

--- Page 2 ---

The present feasibility study aims to demonstrate the use of open-source LLMs in clinical settings for secure and efficient analysis of text-based CMR imaging reports; By evaluating the performance of locally deployed LLMs on these reports, we showcase an alternative to cloud-based solutions, which often raise privacy concerns when handling sensitive patient data.

--- Page 3 ---

Figure 3 shows the execution time (seconds per patient) for each LLM evaluated and average time for our cardiologist evaluator; The results reveal significant differences in computational efficiency.

--- Page 4 ---

Figure 4. The proposed pipeline for analyzing imaging reports to automatically extract diagnostic information from the text, including three steps to 1) split clinical reports into a first quantitative and a second narrative information, 2) translate the narrative text from German to English, and 3) analyze the English text with open-source locally-run LLMs to infer the diagnosis.

--- Page 5 ---

Figure 2. depicts confusion matrices illustrating model performance and human performance in classifying eight cardiac diagnostic categories from unstructured imaging reports; Strong diagonal dominance in Gemma2-27b, DeepSeek-R1-32B, and Qwen2-5-32B, indicates high accuracy, with occasional misclassifications observed for myocarditis (MYO).

--- Page 6 ---

Figure 3. shows the execution time (seconds per patient) for each LLM and human evaluator, highlighting significant efficiency differences; Smaller models like LLaMA 3 (2.3B) and Mistral were the fastest, while larger models, such as DeepSeek-R1 (70B), had the highest execution time without a clear accuracy advantage.

--- Page 6 ---

performances above 0.93 across most evaluation metrics, highlighting their effectiveness and usefulness in automating the analysis of imaging reports.

--- Page 6 ---

The top four LLMs outperformed our board-certified cardiologist across all evaluation metrics in analyzing and classifying CMR reports; Notably, larger models like DeepSeek-R1 (32B) and qwq required significantly higher execution times (26.37s and 28.64s), with DeepSeek-R1 (70B) recording the highest at 50.77 seconds per patient; Our evaluator spent a similar time of 23.79 seconds per patient.

--- Page 6 ---

Smaller models like LLaMA 3 (2.3B) and Mistral were the fastest, while larger models, such as DeepSeek-R1 (70B), had the highest execution time without a clear accuracy advantage; performances above 0.93 across most evaluation metrics, highlighting their effectiveness and usefulness in automating the analysis of imaging reports.

--- Page 6 ---

The top four LLMs outperformed our board-certified cardiologist across all evaluation metrics in analyzing and classifying CMR reports; Notably, larger models like DeepSeek-R1 (32B) and qwq required significantly higher execution times (26.37s and 28.64s), with DeepSeek-R1 (70B) recording the highest at 50.77 seconds per patient; Our evaluator spent a similar time of 23.79 seconds per patient; The results reveal significant differences in computational efficiency.

1. The improved system can perform automated, secure diagnostic categorization of clinical CMR imaging reports by utilizing locally deployed open-source LLMs to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports.

2. The system achieves high performance in classification, as evidenced by the finding that Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories, with Google’s Gemma 2 model achieving the highest average F1 score of 0.98.

3. The system can be optimized for real-time clinical use because it can select models based on speed, noting that The fastest model, LLaMA 3 (2.3B), had an average execution time of 0.41 seconds per patient.

4. The system provides a mechanism to balance complexity and efficiency by identifying that DeepSeek-R1 (70B) that had the highest execution time, performed worse in classification accuracy, indicating that increased computation does not necessarily improve performance.

Purpose: We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extract diagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. Materials and Methods: We evaluated nine open-source LLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptive findings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy, precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models. Results: Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories. Google's Gemma2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32B with F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistral and DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 score of 0.94) across all evaluation metrics in analyzing CMR reports. Conclusion: Our findings demonstrate the feasibility of implementing open-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fast and resource-efficient diagnostic categorization.