MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis

arXiv:2610.20632 · cs.CL, cs.CV · Submitted 2026-09-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis".

Jane: The gist MeddeX presents a system that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Generation (RAG) and Gemini Vision for evidence-backed medical report analysis.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper titled "MeddeX: A Retrieval-Augmented LLaMA-three System with Gemini Vision for Evidence-Backed Medical Report Analysis <ref:2610.20632#pg1,MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence>." It looks like they've put a QLoRA fine-tuned LLaMA-three model together with Retrieval Augmented Generation and Gemini Vision to analyze medical reports <ref:2610.20632#pg1,a QLoRA fine-tuned LLaMA-3>.

Jane: That sounds pretty complex. So what's the main goal here, Tom? What problem are they trying to solve with this system?

Tom: The whole idea is tackling the slow, error-prone, and hard-to-scale manual review of modern medical reports. They combine a fine-tuned LLaMA-three model with RAG and Gemini Vision to give accurate summaries that actually stick to evidence in the report <ref:2610.20632#pg1>.

Lu: It's interesting how they are moving beyond just using a standard LLM for summarization without any source grounding, which is something they say is limiting clinician trust <ref:2610.20632#pg2>.

Meng: So, what does this actually mean for a doctor sitting at their desk? Is this going to be something they can just plug into their existing workflow right now?

Tom: Well, the system handles different kinds of inputs—text PDFs, scanned image-based ones, and even direct radiological images. It uses a pipeline that checks for corrupted text first before sending it to the vision layer if needed <ref:2610.20632#pg1>.

Jane: And what about the authors? It lists Rajdeep Pandhare, Rohit Kundaikar, Yakub Kojar, and Shreyansh Tiwari from the Department of Artificial Intelligence and Data Science at Rajiv Gandhi Institute of Technology.

Lu: They're coming from a place where they are working on model training and representation learning <ref:2610.20632#pg3>. Their focus seems to be on how to make these grounded outputs reliable for clinical use.

Meng: Reliability is key, but running a system like this at scale sounds resource intensive. How big is this thing, and what's the immediate deployment plan?

Tom: They say they reduced VRAM requirements for fine-tuning by approximately four times compared to full BF16 finetuning <ref:2610.20632#pg3>, which helps with the cost of training. And they use QLoRA, which is a technique for making large models more efficient.

Jane: So the authors are showing that you can get strong performance by combining these different AI components rather than just relying on one thing.

Lu: Yes, it’s about showing that integrating different modalities and techniques leads to better results when you need verifiable answers <ref:2610.20632#pg2>.

The paper's summary: Tom: Okay, so the core of MeddeX is this hybrid extraction pipeline. It starts with ingestion, then garbage detection to spot bad text, then smart extraction routing pages to either standard PDF tools or Gemini Vision OCR if they look corrupted <ref:2610.20632#pg1>.

Jane: That garbage detection sounds really clever for handling messy medical documents. Can you explain what triggers that check?

Tom: It checks for things like UTF-sixteen hex artifacts or high ratios of non-printable characters in the text, and if it sees those, it reroutes that page to Gemini Vision OCR instead of just trying to read the corrupted text <ref:2610.20632#pg1>.

Meng: If you're running this in a production environment, what happens when the garbage detection fails or flags a lot of pages as bad? Does it just fail, or does it have a fallback?

Tom: It reroutes those corrupted pages to Gemini Vision OCR for extraction, which is why they call it smart multimodal extraction <ref:2610.20632#pg1>. So the system tries multiple paths until it gets clean text.

Jane: And once they get the text extracted, how do they actually use that information to give an answer? That's where the RAG comes in, right?

Tom: Exactly. They chunk that extracted text into small pieces, like five hundred characters long with a fifty character overlap to keep local context <ref:2610.20632#pg3>. Then they embed those chunks using all-MiniLM-L6-v2 and store those embeddings in Supabase <ref:2610.20632#pg3>.

Lu: Storing three hundred eighty-four dimensional vectors is a common approach for dense retrieval, but they are using that to enable report scoped retrieval later, which I think is the most important part for clinical accuracy <ref:2610.20632#pg3>.

Jane: So when a user asks a question about a specific report, the system doesn't search everything; it narrows down the chunks to only those related to that active report <ref:2610.20632#pg3>. That seems very smart for keeping things focused.

Tom: It does. Then they retrieve the top-k most similar chunks using an RPC called match chunks by report, and then they feed those retrieved passages along with the chat history into the fine-tuned LLaMA-three model for generation <ref:2610.20632#pg3>.

The paper's improvements: Tom: They implemented a few specific technical mechanisms that make this system robust. First, they have this garbage-text detection function, is garbage text, which guards against silently corrupted text extractions <ref:2610.20632#pg1>.

Jane: That sounds like a safety net for the extraction process. What else are they focusing on improving in the actual generation part? Because that's where we need to know if it actually gives us good answers.

Tom: They use a QLoRA fine-tuned LLaMA-three model, specifically the checkpoint LLaMA3-Med42-8B, which they say shows consistent improvements over their base model across several metrics <ref:2610.20632#pg4>.

Lu: The results they show are pretty solid. They report gains across ROUGE-one ROUGE-two and ROUGE-L scores, getting up to +eleven point eight percent in the BLEU score <ref:2610.20632#pg4>. That indicates the fine-tuning is making it much better at generating precise phrases compared to just using the base LLaMA3-Med42-8B <ref:2610.20632#pg4>.

Meng: If you're looking at deployment, what about speed? When a doctor asks a question, how fast is this whole process from upload to answer? We need low latency for clinical tools.

Tom: They use nucleus sampling with top-p set to zero point nine and temperature set to zero point three for Q andA, which keeps the output focused and somewhat consistent <ref:2610.20632#pg4>. And they keep the maximum number of new tokens for Q andA at three hundred with a larger limit of six hundred for report comparisons <ref:2610.20632#pg4>.

Jane: That’s helpful detail on the generation side. It sounds like they are balancing being creative enough to generate a good answer with being constrained enough to stay grounded in the retrieved chunks.

Lu: They also explicitly mention that they avoid hallucination by conditioning the LLaMA-three generator on those retrieved passages, which is what their RAG objective defines as Eq <ref:2610.20632#pg1>. one <ref:2610.20632#pg2>.

Meng: So, the main improvement here seems to be making sure that when it does generate something, it has a traceable source for every claim. That's a big deal for clinical adoption if it’s not just guessing at facts.

Conclusion: Tom: To wrap up MeddeX, this paper presents a retrieval-augmented LLaMA-three system with Gemini Vision that handles various medical document types and aims to provide evidence-backed summaries <ref:2610.20632#pg1,a retrieval-augmented LLaMA-3 system with Gemini Vision>. The main contribution is the integrated pipeline combining garbage detection, smart extraction, chunking with embeddings in Supabase, and report-scoped retrieval <ref:2610.20632#pg3>.

Jane: It’s a system that’s designed for clinical utility by ensuring the generated answers can be traced back to specific source chunks in the original reports <ref:2610.20632#pg3>. This moves us closer to having tools that clinicians can actually trust with their patients' information.

Lu: The work shows how combining domain pretraining, like BioBERT two or Longformer four, with a RAG framework allows for outputs to be traced to source evidence <ref:2610.20632#pg2>. It’s about grounding the language model in the actual documents.

Meng: From an engineering standpoint, what's the main constraint they acknowledge? They mention that for the comparison endpoint, they currently truncate each report to one thousand five hundred characters of context for the prompt <ref:2610.20632#pg4>. That’s a limitation we have to keep in mind when deploying it.

Tom: Right, so it handles a lot—text and images—and shows strong gains over the base model, especially in BLEU score improvements of seventy-one point four percent <ref:2610.20632#pg4>. It’s a production-oriented system built with efficiency in mind through QLoRA and VRAM reductions <ref:2610.20632#pg3>.

Jane: So, the big picture is that they are building a comprehensive system for analysis, not just a single piece of software, by focusing on how the retrieval mechanism connects to the generation step <ref:2610.20632#pg4>.

Lu: They also touched on future work involving model distillation to support inference on edge servers or low-VRAM GPUs for resource-constrained settings <ref:2610.20632#pg4>. That shows they are thinking about how to make this accessible beyond big data centers.

Meng: And I'm looking forward to seeing if they can do that real-time summarization inside actual electronic health record systems, which would really change the workflow for doctors one <ref:2610.20632#pg2>.

Tom: Exactly. So the MeddeX paper is a solid example of how we can take established models and build complex, evidence-backed tools by adding smart retrieval and multimodal capabilities to solve real problems in healthcare <ref:2610.20632#pg1>.

Rajiv Gandhi Institute of Technology

cs.CL, cs.CV

Submitted: 2026-09-30

Updated: 2026-09-30

Comments: 6 pages , 3 figures, published in ijsrem

Journal ref: IJSREM, International Journal of Scientific Research in Engineering and Management Volume 10 | 4| April 2026

License: http://creativecommons.org/publicdomain/zero/1.0/

Importance score: 88/100

The gist: The gist MeddeX presents a system that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Generation (RAG) and Gemini Vision for evidence-backed medical report analysis.

Key concepts

QLoRA Fine-tuning: This is a method used to adapt a large language model (LLaMA-3) for specific tasks efficiently. Instead of retraining the entire massive model, QLoRA allows for fine-tuning using significantly less memory and computational power, making it practical for specialized medical applications.
Retrieval-Augmented Generation (RAG): RAG enhances an AI's response capability by allowing it to look up relevant information from a specific database before generating an answer. In MeddeX, this means the model retrieves precise text chunks from uploaded reports to ensure its summaries are grounded in the actual document content.
Gemini Vision Layer: This component allows the system to process not just text but also visual data, such as scanned PDFs and radiological images. It acts as an intelligent OCR engine that extracts information from images, ensuring that all types of medical documents can be analyzed effectively.

Terminology

Fine-Tuning — 1× in this paper · explained in 5 episodes
Grounding — 1× in this paper · explained in 3 episodes
RAG — 1× in this paper · explained in 3 episodes

Summary

The gist

MeddeX presents a system that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Generation (RAG) and Gemini Vision for evidence-backed medical report analysis.

System Overview

MeddeX is designed to address the limitations of existing medical report analysis by integrating several advanced components into a unified system The system handles text-native PDFs, scanned imagebased PDFs, and direct radiological images through an intelligent hybrid extraction pipeline It features a FastAPI backend with Supabase vector store and a Next.js frontend to complete the end-to-end architecture The system exposes three primary endpoints: /upload report, /consult, and /compare reports

Processing Pipeline

The end-to-end pipeline follows a structured sequence of operations to ensure robust document handling The steps include:

Ingestion, where a user uploads a PDF or image via the frontend and the backend receives the file through the /upload report endpoint.
Garbage Detection, which evaluates each page’s text by calling is garbage text for UTF16 hex artifacts or high non-printable character ratios, rerouting corrupted pages to Gemini Vision
Smart Extraction, which routes text-native pages through PyPDF2 and scanned or garbled pages through Gemini Vision OCR
Chunking, where extracted text is split into overlapping chunks of 500 characters with a 50-character overlap to preserve local context
Embedding & Indexing, where each chunk is encoded using all-MiniLM-L6-v2 and stored with its 384-dimensional embedding vector in Supabase
Report-Scoped Retrieval, which at query time retrieves the top-k most similar chunks via match chunks by report RPC, scoping results to the active report
Generation, where retrieved chunks plus recent chat history are concatenated and passed to the finetuned LLaMA-3 model to generate an evidencegrounded answer
Response & Persistence, where the answer is returned and stored in the chats table with a report id for session continuity

Key Technical Components

Several novel mechanisms enhance MeddeX's robustness and clinical utility

** is garbage text function:**

This function guards against silently corrupted text extractions from UTF-16-encoded or otherwise malformed PDFs by detecting long hex runs or high ratios of non-printable characters, flagging pages for re-routing to Gemini Vision OCR

** Smart Multimodal Extraction:**

This module utilizes a routing logic where files ending in image extensions are processed directly by Gemini Vision OCR, while text pages are only accepted if they pass the garbage detection check and have sufficient length

Retrieval and Generation

MeddeX implements a Retrieval-Augmented Generation (RAG) objective defined in Eq. 1, where the retriever uses dense embeddings to score chunk relevance The generation module uses a QLoRA fine-tuned checkpoint of LLaMA3-Med42-8B, conditioned on the concatenated top-k retrieved passages Inference uses nucleus sampling (top-p = 0.9, temperature = 0.3) with a maximum of 300 new tokens for Q&A and 600 for report comparisons

Longitudinal Analysis and Evaluation

MeddeX introduces features critical for clinical workflow improvement

** Report-Scoped Retrieval:**

This feature enables retrieval to be scoped to a specific document via the report id parameter, preventing context bleed across multiple uploaded reports

** Longitudinal Report Comparison:**

The /compare reports endpoint retrieves full text for two reports and generates a comprehensive summary covering parameter-by-parameter changes, clinically significant findings, stable values, and an overall health trend

Evaluation against the base LLaMA3-Med42-8B model demonstrates consistent improvements across ROUGE-1 (+9.2%), ROUGE-2 (+10.0%), ROUGE-L (+11.8%), BLEU (+71.4%), and BERTScore (+0.8%) The most pronounced gain is in the BLEU score, indicating that fine-tuning substantially improves the model’s ability to generate precise, phrase-level matches to reference clinical answers QLoRA fine-tuning reduces VRAM requirements for fine-tuning by approximately 4× compared to full BF16 finetuning and inference VRAM is similarly reduced The system's contributions position MeddeX as a comprehensive, production-oriented clinical AI system.

Future Work

Planned extensions include large-scale clinical validation across specialties and languages to assess generalizability Other planned work involves hierarchical summarization for comparison to replace fixed 1,500-character truncation and integrating real-time summarization within electronic health record systems The system also plans model distillation to support inference on edge servers or low-VRAM GPUs for resource-constrained settings The authors also plan to add explainability through calibrated confidence scores and claimlevel source attribution surfaced directly in the frontend UI Finally, expanded multimodal support is planned to include end-to-end radiology report generation from raw imaging inputs The paper concludes by stating that occasional hallucinations on rare terminology were observed and the comparison endpoint currently truncates each report to 1,500 characters of context for the prompt. The system's design choices reflect a deliberate preference for factual consistency over generative diversity in clinical contexts The system's contributions position MeddeX as a comprehensive, production-oriented clinical AI system.

ACKNOWLEDGMENT

The authors sincerely thank Dr. Sanjay U. Bokade (Principal) and Dr. Jyoti Deshmukh (H.O.D., Department of AI & Data Science), Rajiv Gandhi Institute of Technology, for providing the opportunity to undertake this project Special thanks to Prof. Naresh Shende for continuous guidance and encouragement throughout the work.

REFERENCES

[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186

[2] J. Lee et al., “BioBERT: A pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020

[3] M. Lewis et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in Proc. ACL, 2020, pp. 7871–7880

[4] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The LongDocument Transformer,” arXiv:2004.05150, 2020<ref:2610.

Improvements for AI systems

textbf Rewritten Prompt Context for Comparison Engine Fields: The current prompt for /compare reports is limited by a fixed context window, which may omit critical details from very long documents; future work will address this with hierarchical summarization. This can be improved by replacing the fixed 1,500-character truncation with a multi-pass summarization strategy to ensure all necessary parameter comparisons are captured regardless of document length.
textbf Enhanced Explainability and Confidence Scoring: The paper mentions future work including Calibrated confidence scores and claimlevel source attribution surfaced directly in the frontend UI. Implementing this would allow clinicians to trust the output by seeing not just the summary, but also a visual indicator of where each sentence's information is grounded in the original report text.
textbf Real-Time EHR Integration: Integrating MeddeX as a real-time summarization tool within Electronic Health Record (EHR) systems would allow for immediate clinical decision support at the point of care, moving beyond batch processing to continuous monitoring. This capability would enable clinicians to query patient history instantly with evidence-backed answers derived from their own records.
textbf Hierarchical Summarization for Comparison: To address the limitation noted in the discussion section regarding long reports, developing a multi-pass summarization strategy for the /compare reports endpoint would allow the system to generate a summary that captures parameter changes across multiple visits without losing detail due to fixed context window constraints.
textbf Edge Deployment via Model Distillation: Distilling the QLoRA fine-tuned model to support inference on edge servers or low-VRAM GPUs would allow MeddeX to be deployed on resource-constrained clinical workstations, ensuring low-latency responses even in environments without high-end enterprise hardware.

Abstract

Modern healthcare generates increasingly complex medical reports that challenge clinical professionals in terms of time and interpretive burden. Manual review processes are slow, error-prone, and difficult to scale. This paper presents MeddeX, a Medical Report Analyzer that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Gen eration (RAG) and a multimodal Gemini Vision layer to deliver accurate, concise, and evidence-backed summaries of medical documents. The system handles text-native PDFs, scanned image based PDFs, and direct radiological images through an intelligent hybrid extraction pipeline that includes a novel garbage-text detection mechanism for corrupted and UTF-16-encoded doc uments. Beyond single-report Q&A, MeddeX introduces report scoped retrieval for precise contextual grounding, an automated longitudinal report comparison engine, and a self-healing backfill pipeline that lazily processes legacy documents on demand. A FastAPI backend, Supabase vector store, and Next.js fron tend complete the end-to-end architecture. Evaluation against the base LLaMA3-Med42-8B model demonstrates consistent improvements across ROUGE-1 (+9.2%), ROUGE-2 (+10.0%), ROUGE-L (+11.8%), BLEU (+71.4%), and BERTScore (+0.8%), validating the benefit of domain-specific fine-tuning. MeddeX represents a practical and robust step toward AI-assisted clinical decision support.