Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?

arXiv:2606.15412 · cs.CL, cs.AI · Submitted 2026-06-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Few-Shot Biomedical Relation Extraction with Large Language Models".

Jane: Here's the gist of the paper. The gist The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56) Biomedical relation extraction (BioRE) is a key step in transforming biomedical literature into structured knowledge Most existing approaches rely on supervised models trained on costly annotated datasets, limiting their scalability and adaptability across relation types and domains This has motivated interest in prompt-based methods with large language models (LLMs) that are better suited to low-resource settings and cross-domain generalization LLMs encode extensive linguistic and factual knowledge in their parameters, which can be leveraged during inference The combination of learned knowledge and prompt-conditioned generation enables LLMs to perform a wide range of tasks without task-specific training Related Work and Challenges Rule-based methods rely on co-occurrence heuristics or manually designed rules based on lexical and syntactic patterns While some achieve high precision, these methods are typically laborintensive and suffer from low recall and limited generalizability Modern supervised approaches predominantly rely on models trained to predict relations from annotated examples PLM-based methods typically follow a two-stage paradigm, where models are pre-trained on large unlabeled corpora and then fine-tuned on task-specific datasets However, these models are computationally expensive to train and generalize poorly when trained on small or highly specialized datasets Evaluation Setup and Models Experiments were conducted on the BioREDirect dataset The dataset consists of PubMed abstracts hand-annotated with six entity types and eight relation types The relation type distribution is highly imbalanced, with Association, Positive Correlation, and Negative Correlation accounting for more than 95% of all relation instances Models employed include recent open-source LLM families like Gemma-4 and Qwen-3.5 The evaluation focused on document-level BioRE, which is more challenging due to the larger number of entity pairs and long-range dependencies Task Formulations and Performance The study compared two task formulations: pairwise classification, where the model predicts relations for individual entity pairs at a time, and joint generation, where the model extracts multiple relations in a single model call Pairwise classification achieves higher recall but produces more false positives by collapsing multi-step biological pathways into direct pairwise relations Joint generation adopts a more conservative strategy, typically extracting only the most salient relations, which yields higher precision at the expense of recall The best results are shown in bold in Table 2 for both task formulations Key Findings on Relation Types and Efficiency The Association class is inherently ambiguous, serving as a fallback label for cases where a more specific relation type cannot be determined confidently during annotation The LLM-based approach frequently assigned more specific mechanistic relation types, such as Positive, Negative, or Bind The model outperforms the supervised baseline on several low-frequency relation types, including Bind and Cotreatment When considering macro-F1 scores, the performance gap narrows considerably, with the best result achieved by Gemma-4 models in the generation setting Joint generation provides a substantial efficiency advantage, reducing computational costs by up to 25× compared to pairwise classification The proposed k-constrained generation setting provides an effective intermediate alternative for balancing precision, recall, and inference cost LLMs provide a viable alternative to supervised systems, particularly in settings with limited annotated data and well-defined relation semantics Pairwise classification is better suited for applications requiring maximal relation coverage, while joint generation is attractive when extraction quality and efficiency are prioritized Future work could involve larger and biomedical domain-specific LLMs, uncertainty-aware prediction strategies, and advanced prompting techniques like chain-of-thought reasoning The study also noted that evaluation focused exclusively on EP and RT extraction, suggesting that incorporating relation directionality or novelty would provide a more comprehensive assessment of BioRE performance Data and Code Availability is available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression) The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore <ref:

Paper summary: Tom: To unpack what they actually did, the paper looks at how prompt strategies affect BioRE performance across those two task formulations, pairwise classification and joint generation. They use open-weight models like Gemma-four and Qwen-three point five for this investigation <ref:2606.15412#pg3,Gemma-4 and Qwen-3.5>.

Jane: The abstract tells us that for the BioREDirect dataset, pairwise classification achieves higher recall, but joint generation is more precise and computationally efficient when you look at it in terms of accuracy per call.

Lu: They also found that the best-performing model achieved a micro-F1 score of zero point four four, which they say substantially outperforms previous few-shot results that were around zero point three four, even though it's still below the supervised baseline score of zero point five six > <ref:2606.15412#pg1>, page one of that work reads: "Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?".

Meng: That gap between the model and the supervised baseline is something I'm paying attention to. They attribute much of that difference to a single relation type being ambiguously defined in their annotation process.

Lalam: And when they look at macro-F1 scores, which gives a better picture across different relation types because some are really rare, the prompt-based approaches still manage to outperform the supervised baseline with a score of zero point four five compared to zero point three eight > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Tom: It’s interesting how they found that the Association class is inherently ambiguous, acting like a fallback label when the model can't be sure about a more specific relation type during annotation.

Jane: And for those rare types, like Bind or Cotreatment, the LLM approach frequently assigned more specific mechanistic relation types such as Positive or Negative relations when it compared to the supervised baseline > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Lu: The paper also suggests that joint generation provides a substantial efficiency advantage, reducing the computational costs by up to twenty-five times compared to pairwise classification > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Meng: Twenty-five times is a big number if we’re talking about inference costs. I'm interested in the proposed k-constrained generation setting because it sounds like an effective way to balance precision, recall, and that high inference cost > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Lalam: So to wrap up this summary, the paper is testing prompt-based learning with LLMs against supervised methods for few-shot BioRE, finding a trade-off between recall and precision based on whether you do pairwise classification or joint generation > <ref:2606.15412#pg1>, page one of that work reads: "Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?".

Conclusion: Tom: So, looking at the title, "Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?", the authors are suggesting that LLMs offer a practical way forward when we don't have those huge annotated datasets we usually need.

Jane: They’re arguing that prompt-based learning with these large language models is a viable alternative because it works well in low-resource settings and can generalize across different relation types and domains, which was the main problem with traditional supervised learning.

Lu: The implication here is that we don't necessarily need to train massive models from scratch for every specific biomedical task if we can leverage the existing vast knowledge encoded within a large language model through just careful prompting > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Meng: From a deployment standpoint, this suggests that for many applications where annotated data is scarce or expensive to get, leveraging an existing LLM via prompting is a much more feasible path than building and fine-tuning a custom supervised model > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Lalam: The study points out that while pairwise classification is better for getting all the relations covered, joint generation is preferred when you need high quality and efficiency in a single output, and it highlights the importance of having well-defined relation schemas for this approach to really shine > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Tom: It boils down to the fact that while supervised models are still strong—hitting a micro-F1 score of zero point five six on the baseline for comparison—the prompt-based approach shows promise in reaching performance levels close to that using much less training effort > <ref:2606.15412#pg3>, page three of that work reads: "Few-Shot BioRE with LLMs: A Viable Alternative to Supervised Learning? three ence <ref:2606.15412#pg1>."

Jakob Mraz, Tomaž Curk, Blaž Zupan

University of Ljubljana · Baylor College of Medicine

cs.CL, cs.AI

Submitted: 2026-06-13

Updated: 2026-08-03

DOI: 10.1007/978-3-032-41086-3_14

Code: https://github.com/jkbmrz/few-shot-biore

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 75/100

The gist: The gist The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56) Introduction

Key concepts

Biomedical Relation Extraction (BioRE): This is the process of automatically identifying relationships between entities mentioned in biomedical text, like genes and diseases. It turns raw scientific literature into structured knowledge by labeling how different biological concepts connect, which is crucial for understanding medical information.
Few-Shot Learning: This technique involves training a model to perform a new task with only a very small number of examples (a 'few shots) rather than needing thousands. In this paper, LLMs are used to learn the required relation extraction patterns from just a few examples, making it adaptable without extensive retraining.
Joint Generation vs. Pairwise Classification: These are two ways to ask an LLM for answers: pairwise classification asks the model to check every pair of entities individually, which is good for finding many relations but can create errors. Joint generation asks the model to extract several relations all at once in one go, leading to higher precision and saving computational time.
Micro-F1 Score: This is a performance metric that calculates the harmonic mean of precision and recall across all classes. It's used here to get a single, overall measure of how well the model performs on all relation types combined, which helps compare different models fairly.

Terminology

Adaptability — 1× in this paper · explained in 1 episode
F1 Score — 1× in this paper · explained in 4 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56)

Introduction and Motivation

Biomedical relation extraction (BioRE) is a key step in transforming biomedical literature into structured knowledge Most existing approaches rely on supervised models trained on costly annotated datasets, limiting their scalability and adaptability across relation types and domains This has motivated interest in prompt-based methods with large language models (LLMs) that are better suited to low-resource settings and cross-domain generalization LLMs encode extensive linguistic and factual knowledge in their parameters, which can be leveraged during inference The combination of learned knowledge and prompt-conditioned generation enables LLMs to perform a wide range of tasks without task-specific training

Related Work and Challenges

Rule-based methods rely on co-occurrence heuristics or manually designed rules based on lexical and syntactic patterns While some achieve high precision, these methods are typically laborintensive and suffer from low recall and limited generalizability Modern supervised approaches predominantly rely on models trained to predict relations from annotated examples PLM-based methods typically follow a two-stage paradigm, where models are pre-trained on large unlabeled corpora and then fine-tuned on task-specific datasets However, these models are computationally expensive to train and generalize poorly when trained on small or highly specialized datasets

Evaluation Setup and Models

Experiments were conducted on the BioREDirect dataset The dataset consists of PubMed abstracts hand-annotated with six entity types and eight relation types The relation type distribution is highly imbalanced, with Association, Positive Correlation, and Negative Correlation accounting for more than 95% of all relation instances Models employed include recent open-source LLM families like Gemma-4 and Qwen-3.5 The evaluation focused on document-level BioRE, which is more challenging due to the larger number of entity pairs and long-range dependencies

Task Formulations and Performance

The study compared two task formulations: pairwise classification, where the model predicts relations for individual entity pairs at a time, and joint generation, where the model extracts multiple relations in a single model call Pairwise classification achieves higher recall but produces more false positives by collapsing multi-step biological pathways into direct pairwise relations Joint generation adopts a more conservative strategy, typically extracting only the most salient relations, which yields higher precision at the expense of recall The best results are shown in bold in Table 2 for both task formulations

Key Findings on Relation Types and Efficiency

The Association class is inherently ambiguous, serving as a fallback label for cases where a more specific relation type cannot be determined confidently during annotation The LLM-based approach frequently assigned more specific mechanistic relation types, such as Positive, Negative, or Bind The model outperforms the supervised baseline on several low-frequency relation types, including Bind and Cotreatment When considering macro-F1 scores, the performance gap narrows considerably, with the best result achieved by Gemma-4 models in the generation setting Joint generation provides a substantial efficiency advantage, reducing computational costs by up to 25× compared to pairwise classification The proposed k-constrained generation setting provides an effective intermediate alternative for balancing precision, recall, and inference cost

Conclusion and Future Work

LLMs provide a viable alternative to supervised systems, particularly in settings with limited annotated data and well-defined relation semantics Pairwise classification is better suited for applications requiring maximal relation coverage, while joint generation is attractive when extraction quality and efficiency are prioritized Future work could involve larger and biomedical domain-specific LLMs, uncertainty-aware prediction strategies, and advanced prompting techniques like chain-of-thought reasoning The study also noted that evaluation focused exclusively on EP and RT extraction, suggesting that incorporating relation directionality or novelty would provide a more comprehensive assessment of BioRE performance

Data and Code Availability is available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression) The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore Funding was supported by core funding P2-0209 (Artificial Intelligence and Intelligent Systems) and project grants GC-0001 (Artificial Intelligence for Science) and L2-60154 (Explainable Foundation Models for Human Gene Expression), all funded by Slovenian Research and Innovation Agency The source code and input data used in this study are available at https://github.com/jkbmrz/few-shot-biore <ref:

Improvements for AI systems

Pairwise classification models can be improved by incorporating uncertainty-aware prediction strategies that leverage token probabilities to filter low-confidence predictions in the classification setting. This allows for a more nuanced handling of ambiguous or difficult entity pair relations.
Joint generation models can be enhanced by utilizing advanced prompting techniques such as chain-of-thought reasoning to improve performance on complex tasks at the cost of increased inference latency. This enables the model to generate intermediate reasoning steps before producing a final response, potentially improving accuracy for intricate biological interactions.
Incorporate task-specific preprocessing methods like entity masking, synonym replacement, and irrelevant entity filtering into the prompt design for both classification and generation tasks. These methods can help guide the LLM towards extracting more specific or relevant relations by refining the input text before it is processed.
Future work should include assessing relation directionality and novelty in BioRE performance metrics to provide a more comprehensive evaluation of LLMs, as these aspects are currently omitted from the study's focus. This would offer a richer understanding of the models' capabilities across all required relation types in the dataset.

Abstract

Biomedical relation extraction (BioRE) is a key step in transforming biomedical literature into structured knowledge. Most existing approaches rely on supervised models trained on costly annotated datasets, limiting their scalability and adaptability across relation types and domains. We investigate few-shot BioRE using prompt-based learning with large language models (LLMs) and compare two task formulations: pairwise classification, which predicts relations for individual entity pairs, and joint generation, which extracts multiple relations in a single model call. Experiments on the BioREDirect dataset reveal a clear precision-recall trade-off. Pairwise classification achieves higher recall, whereas joint generation is more precise and computationally efficient. The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56). Much of this gap is attributable to a single ambiguously defined relation type. When evaluated using macro-F1, which better captures performance across relation types in an imbalanced setting, prompt-based approaches outperform the supervised baseline (0.45 vs. 0.38), particularly on rare relation types. These findings highlight the potential of LLMs for BioRE in low-resource settings and underscore the importance of well-defined relation schemas.

Sources

Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier