Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

arXiv:2607.13430 · cs.CL · Submitted 2026-07-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation".

Jane: Here's the gist of the paper. The gist The study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks, exploring post-training methods like SFT, DPO, ORPO, and GRPO with Qwen-based SLMs on medication package leaflets to assess cross-dataset generalizability.

Tom: Dataset and Models The study uses a D2T dataset focusing on medication leaflets - BioLeaflets curated from a corpus of 1,336 package leaflets of medicines sampled from the European Medicines Agency website<ref:2607.13430#pg2> The dataset is split into six sections, including product overview, pre-administration information, dosage instructions, side effects, storage instructions, and package & other information identified by heuristic rules<ref:2607.13430#pg3> To test model generalizability, a new drug label dataset called FDALeaflets is constructed using drug labeling information collected from openFDA<ref:2607.13430#pg3> This new dataset reflects distinct regulatory conventions, section taxonomies, and linguistic patterns compared to BioLeaflets<ref:2607.13430#pg3> The base models employed are a family of Qwen2.5-based SLMs evaluated across three model sizes—0.5B, 3B, and 7B parameters<ref:2607.13430#pg3> They also compare against Meditron3-Qwen2.5-7B, a medically fine-tuned variant of Qwen<ref:2607.13430#pg3> Baselines included the BioLeaflet T5 model and GPT-5<ref:2607.13430#pg3>.

Jane: Post-Training Alignment Methods The researchers systematically evaluate four representative post-training alignment strategies: SFT, DPO, ORPO, and GRPO<ref:2607.13430#pg4> Supervised fine tuning (SFT) is the most widely used approach for domain adaptation, where models are trained on paired input–output examples by optimizing the negative log-likelihood of a reference text conditioned on the input<ref:2607.13430#pg5> Prompt loss weighting (PLW) is introduced to control the relative contribution of prompt tokens to the overall training objective, which is adopted for SFT and ORPO experiments<ref:2607.13430#pg6>.

Tom: Direct preference optimization (DPO) optimizes a language model using pairs of chosen and rejected responses without requiring an explicit reward model<ref:2607.13430#pg6> Odds ratio preference optimization (ORPO) extends preference-based alignment by unifying supervised likelihood maximization and preference learning within a single training objective<ref:2607.13430#pg6> Group relative policy optimization (GRPO) is a reinforcement learning–based method that optimizes a policy using relative preferences among groups of generated outputs rather than isolated preference pairs<ref:2607.13430#pg6>.

Paper summary: Tom: So looking at the full picture of "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet," we see that the work systematically compared SFT, DPO, ORPO, and GRPO using Qwen2 point 5 models on medication leaflets <ref:2607.13430#pg1>

Jane: The main implication is that choosing the right alignment strategy is central to getting a model that actually performs well in this specialized biomedical data-to-text generation task <ref:2607.13430#pg1>

Lu: The paper establishes that ORPO provides the strongest performance among the RL-free methods for semantic fidelity, while GRPO demonstrates better robustness when moving from training data to new regulatory domains <ref:two thousand six hundred seven point one three four three zero#pg2

Meng: For those of us shipping models, it’s a clear signal that we need to be very deliberate about which fine-tuning objective we choose based on whether our primary goal is maximizing accuracy within the known domain or handling shifts in the regulatory environment <ref:two thousand six hundred seven point one three four three zero#pg2

Tom: Exactly, and it highlights that decoder-only small language models are performing well here, beating encoder–decoder baselines and proprietary LLMs constrained to prompt adaptation <ref:2607.13430#pg10>

Jane: It sets a foundation for how we approach advancing biomedical generative NLP beyond just simple supervised fine-tuning, emphasizing that alignment strategy selection is a central design consideration for deployable systems <ref:two thousand six hundred seven point one three four three zero#pg10

Conclusion: Tom: So we’re wrapping up on this paper about "Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet" and the main authors are from a group focused on applying these models to healthcare documentation tasks <ref:two thousand six hundred seven point one three four three zero#pg1

Jane: It looks like the core idea is testing four different ways—SFT, DPO, ORPO, and GRPO—to fine-tune these small models so they can actually write accurate patient instructions instead of just guessing <ref:two thousand six hundred seven point one three four three zero#pg1

Lu: The paper sets up a comparison between these methods on a standard dataset called BioLeaflets and then tests how well they generalize when the model sees a completely different set of regulatory rules in this new dataset called FDALeaflets <ref:two thousand six hundred seven point one three four three zero#pg2

Meng: I’m curious about the real deployment stuff here. If we’re shipping these models, which one of these alignment methods would I actually trust to handle a real-world scenario where the input data changes slightly <ref:two thousand six hundred seven point one three four three zero#pg2

Tom: Well, what they found is that ORPO performed best in terms of semantic similarity on the original data and it consistently beat SFT and DPO across most metrics <ref:two thousand six hundred seven point one three four three zero#pg7

Jane: That’s interesting because ORPO combines supervised learning with preference learning without needing an explicit reward model, which is a neat trick for making the models better at matching the specific style of medical text <ref:two thousand six hundred seven point one three four three zero#pg7

Lu: But then they look at the generalization—when you take those aligned models and try to use them on that new FDALeaflets set, GRPO actually showed the best performance there <ref:two thousand six hundred seven point one three four three zero#pg8

Meng: So ORPO is good for accuracy in a known setting, but GRPO seems better at handling the shift when regulatory styles change. That’s a big difference for us if we’re dealing with different drug labeling standards across different regions <ref:two thousand six hundred seven point one three four three zero#pg8

Tom: Exactly. They found that GRPO achieved the highest semantic similarity score on that new set, reaching zero point nine four zero seven, which is better than GPT-five and the others tested in this cross-dataset test <ref:two thousand six hundred seven point one three four three zero#pg8

Jane: What this means for us is that we can’t just rely on one alignment technique; you have to pick a strategy based on whether your primary concern is getting perfect text for one specific set of documents or ensuring the model stays robust when it encounters new types of medical paperwork <ref:two thousand six hundred seven point one three four three zero#pg8

Lu: The paper clearly shows that post-training alignment isn't just a small tweak; it’s what separates the models that are actually useful in this specialized biomedical area from the ones that fall short <ref:two thousand six hundred seven point one three four three zero#pg10

Tom: So while the numbers look promising for semantic quality and generalization, the paper makes it clear that getting a model ready for real-world medical use still involves a lot of careful human oversight on top of these alignment techniques <ref:two thousand six hundred seven point one three four three zero#pg10

Xi Yang, Guodong Liu, Chuqin Li, Fan Wu, Ergin Soysal, Min Jiang, Xing He, Jiang Bian, Yi Guo, Shams Zaman, §Thomas Fuchs§Todd Sanger§Yonghui Wu

Advanced Intelligence Eli Lilly and Company Department of Biostatistics and Health Data Science Indiana University Center for Biomedical Informatics Regenstrief Institute Department of Health Outcomes and Biomedical Informatics College of Medicine University of Florida Cancer Informatics Shared Resource University of Florida Health Cancer Center

cs.CL

Submitted: 2026-07-15

Updated: 2026-10-21

Comments: 8 pages, 1 figures

Code: https://github.com/uf-hobi-informatics-lab/BiomedicalSmallLanguageModelAlignment

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 77/100

The gist: The gist The study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks, exploring post-training methods like SFT, DPO,

Key concepts

Small Language Models (SLMs): These are smaller versions of large language models designed to be computationally cheaper and easier to deploy for specific tasks like generating medical text. They are being tested here for their ability to accurately translate structured biomedical data into human-readable patient information.
Post-Training Alignment: This refers to techniques used after a model is initially trained on general data. The researchers tested methods like SFT and ORPO to specifically fine-tune the model's behavior so it generates high-quality, medically accurate text tailored for drug leaflets.
Medication Leaflet Generation: This is the specific biomedical task where a computer model takes structured information about a medicine (like dosage or side effects) and writes a clear, patient-friendly document that explains how to use the drug. Accuracy and readability are crucial here.
Cross-Regulatory Generalization: This tests if a model trained on one set of rules (one drug label format) can still perform well when faced with entirely different rules (a new drug label format). The study used a new dataset, FDALeaflets, to measure this ability.

Terminology

GRPO — 2× in this paper · explained in 6 episodes
Direct Preference Optimization (DPO) — 1× in this paper · explained in 10 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 10 episodes
Semantic Similarity — 1× in this paper · explained in 2 episodes
Small Language Models (SLMs) — 1× in this paper · explained in 5 episodes
Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 13 episodes

Summary

The gist The study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks, exploring post-training methods like SFT, DPO, ORPO, and GRPO with Qwen-based SLMs on medication package leaflets to assess cross-dataset generalizability.

Dataset and Models

The study uses a D2T dataset focusing on medication leaflets - BioLeaflets curated from a corpus of 1,336 package leaflets of medicines sampled from the European Medicines Agency website<ref:2607.13430#pg2> The dataset is split into six sections, including product overview, pre-administration information, dosage instructions, side effects, storage instructions, and package & other information identified by heuristic rules<ref:2607.13430#pg3> To test model generalizability, a new drug label dataset called FDALeaflets is constructed using drug labeling information collected from openFDA<ref:2607.13430#pg3> This new dataset reflects distinct regulatory conventions, section taxonomies, and linguistic patterns compared to BioLeaflets<ref:2607.13430#pg3> The base models employed are a family of Qwen2.5-based SLMs evaluated across three model sizes—0.5B, 3B, and 7B parameters<ref:2607.13430#pg3> They also compare against Meditron3-Qwen2.5-7B, a medically fine-tuned variant of Qwen<ref:2607.13430#pg3> Baselines included the BioLeaflet T5 model and GPT-5<ref:2607.13430#pg3>.

Post-Training Alignment Methods

The researchers systematically evaluate four representative post-training alignment strategies: SFT, DPO, ORPO, and GRPO<ref:2607.13430#pg4> Supervised fine tuning (SFT) is the most widely used approach for domain adaptation, where models are trained on paired input–output examples by optimizing the negative log-likelihood of a reference text conditioned on the input<ref:2607.13430#pg5> Prompt loss weighting (PLW) is introduced to control the relative contribution of prompt tokens to the overall training objective, which is adopted for SFT and ORPO experiments<ref:2607.13430#pg6>. Direct preference optimization (DPO) optimizes a language model using pairs of chosen and rejected responses without requiring an explicit reward model<ref:2607.13430#pg6> Odds ratio preference optimization (ORPO) extends preference-based alignment by unifying supervised likelihood maximization and preference learning within a single training objective<ref:2607.13430#pg6> Group relative policy optimization (GRPO) is a reinforcement learning–based method that optimizes a policy using relative preferences among groups of generated outputs rather than isolated preference pairs<ref:2607.13430#pg6>.

Key Findings on Alignment Strategies

The results show that ORPO outperforms the SFT baselines<ref:2607.13430#pg7> ORPO consistently outperforms both SFT (Table 1) and DPO (Table 3) across all evaluation metrics, with the most pronounced gains observed in semantic textual similarity (STS)<ref:2607.13430#pg7> ORPO increases STS by approximately +0.02 to +0.05 absolute across model sizes<ref:2607.13430#pg7> Prompt loss weighting further strengthens ORPO performance, particularly for larger models, yielding the highest overall scores across all metrics<ref:2607.13430#pg7>. In contrast, DPO underperforms both ORPO and the SFT baseline<ref:2607.13430#pg9> Recent work has identified a phenomenon known as DPO likelihood displacement, where the probabilities of both preferred and rejected outputs can decrease during training, leading to erosion of supervised signals and degradation of semantic fidelity when preference pairs do not exhibit sufficiently distinct differences<ref:2607.13430#pg10>.

Generalizability and Robustness

Models trained on the BioLeaflets dataset are evaluated zero-shot on the FDALeaflets dataset, where all models exhibit a measurable performance decline relative to in-domain evaluation, reflecting the distribution shift<ref:2607.13430#pg8> Among alignment strategies, GRPO demonstrates the strongest generalization behavior<ref:2607.13430#pg8> The Qwen2.5-7B-Inst-GRPO model achieves the highest semantic similarity (0.9407) and METEOR (0.5400), substantially outperforming both GPT-5 (STS of 0.9295; METEOR of 0.5328) and models trained with SFT, DPO, or ORPO<ref:2607.13430#pg8> GRPO improves STS by +0.055 absolute (0.9407 vs. 0.8858), indicating improved semantic robustness under regulatory domain shift<ref:2607.13430#pg8>.

Conclusion

This work provides a systematic evaluation of post-training alignment strategies for SLMs in medication leaflet generation from structured biomedical data<ref:2607.13430#pg10> The findings establish that ORPO offers the strongest in-domain semantic performance among RL-free methods, while GRPO enables superior robustness under regulatory distribution shift<ref:2607.13430#pg10> Full-parameter fine-tuning and prompt-aware training objectives are critical for performance, whereas parameter-efficient fine-tuning remains insufficient for this task<ref:2607.13430#pg10>. Decoder-only SLMs consistently outperform encoder–decoder baselines and proprietary LLMs constrained to prompt-based adaptation<ref:2607.13430#pg10> Together, these findings establish a foundation for advancing biomedical generative NLP beyond supervised fine-tuning and underscore alignment strategy selection as a central design consideration for deployable, privacy-preserving clinical language systems<ref:2607.13430#pg10>.

Discussion on Model Specialization

Across all alignment strategies, the medical pretrained Qwen2.5-7B model, Meditron3-Qwen2.5-7B, does not outperform its non-medical Qwen2.5-7B counterpart<ref:2607.13430#pg10> This suggests that medication leaflet generation is a highly specialized task, and that broad medical-domain pretraining alone may be insufficient<ref:2607.13430#pg10> The Qwen-2.5-7B-Inst-GRPO model, initialized without SFT, exhibits the strongest cross-dataset generalization<ref:2607.13430#pg10> This suggests that while supervised training/initialization promotes specialization to a specific regulatory framework, it may cause overfitting to dataset-specific conventions<ref:2607.13430#pg10> The Qwen2.5-7B-Inst-GRPO model achieves the highest semantic similarity (0.9407) and METEOR (0.5400), substantially outperforming both GPT-5 (STS of 0.9295; METEOR of 0.5328) and models trained with SFT, DPO, or ORPO<ref:2607.13430#pg8>. This suggests that while supervised training/initialization promotes specialization to a specific regulatory framework, it may cause overfitting to dataset-specific conventions<ref:2607.13430#pg10>.

Limitations

The study has limitations as it evaluates model performance using automated lexical and semantic metrics as well as LLM-as-judge approaches on four metrics but does not conduct human evaluation of factuality, safety, or patient comprehension due to resource constraints<ref:2607.13430#pg10> Future work should incorporate expert and lay-user assessments as well to validate clinical reliability and usability<ref:2607.13430#pg10> As a case study, the researchers only explore the medication leaflet generation<ref:2607.13430#pg10> In the future, they plan to extend their study to broader biomedical D2T generation tasks, such as patient narrative generation from structured electronic health records and clinical trial report generation from trial data<ref:2607.13430#pg10>.

Data Availability

Source code is available at https://github.com/uf-hobi-informatics-lab/BiomedicalSmallLanguageModelAlignment FDALeaflets can be found at https://huggingface.co/datasets/UFNLP/FDALeaflet<ref:2607.

Improvements for AI systems

Post-training alignment using Odds Ratio Preference Optimization (ORPO) to generate medication leaflets from structured data will yield higher semantic similarity scores (STS) than Supervised Fine-Tuning (SFT), as ORPO jointly optimizes the odds ratio between preferred and dispreferred outputs while retaining a SFT signal from preferred texts.
Cross-dataset generalization will be enhanced by employing Group Relative Policy Optimization (GRPO), as it demonstrates the strongest generalization behavior and can improve STS by +0.055 absolute when transferring models across regulatory sources like EMA to FDA.
Training small language models (SLMs) on structured biomedical data using full-parameter fine-tuning instead of parameter-efficient fine-tuning will be necessary, as parameter-efficient fine-tuning imposes meaningful limitations compared to full-parameter adaptation due to aggressive parameter constraints.
Integrating Prompt Loss Weighting (PLW) during SFT or ORPO training will improve performance by providing additional gradient signal for tasks with long, information-dense instructions, which is a common characteristic of biomedical data-to-text generation.

Abstract

Translating structured biomedical information into accurate, patient-friendly narratives remains a challenging biomedical data-to-text (D2T) task. While small language models (SLMs) are attractive for healthcare applications because of their lower computational cost, easier deployment, and improved data governance, the impact of post-training alignment on biomedical D2T generation has not been systematically studied. In this work, we compare four alignment methods: supervised fine-tuning (SFT), direct preference optimization (DPO), odds ratio preference optimization (ORPO), and group relative policy optimization (GRPO), using Qwen-2.5 models for medication leaflet generation. We further introduce FDALeaflets, a dataset derived from OpenFDA drug labels, to evaluate cross-regulatory generalization in a zero-shot setting. Selected aligned SLMs outperform GPT-5 on several reference-based metrics, whereas GPT-5 achieves better hallucination and adequacy scores on FDALeaflets under LLM-as-judge evaluation. Among RL-free alignment methods, ORPO achieves the best in-domain semantic performance, reaching a semantic textual similarity (STS) score of 0.9752. GRPO achieves the highest FDALeaflets STS (0.9407), exceeding GPT-5 (0.9295). These findings show that post-training alignment is a key factor in improving both semantic fidelity and generalization for biomedical D2T generation, while clinical safety and patient readability require human verification.