Rethinking Synthetic Rationale Supervision for Disease Prediction

The gist: Supervised fine-tuning with synthetic rationale data consistently and substantially hurts prediction performance relative to label-only fine-tuning for five-year Alzheimer’s disease and related dementias prediction (Page 1).

How it works

Supervised fine-tuning with synthetic rationale data is widely assumed to improve language model performance on clinical prediction tasks by teaching models not just what to predict but why (Page 1). This assumption was tested on five-year Alzheimer’s disease and related dementias (ADRD) prediction from longitudinal health histories across a large-scale controlled experiment of 504 configurations (Page 1). The researchers found that rationale-based SFT consistently and substantially hurts prediction performance relative to label-only fine-tuning (Page 1).

The experimental design involved comparing three training targets: no-rationale, free-rationale, and stepwise-rationale (Page 2). These rationales were generated before SFT from the original training labels using a generator instructed to use only evidence present in the record (Page 2). The controlled SFT grid crossed rationale format, sample size, learning rate, base model (Qwen3-8B and Qwen2.5-7B-Instruct), and decoding setting across 504 configurations (Page 2).

Key Findings on Performance Degradation

Across the controlled sweep, models trained to output only the final label or probability achieved higher ROC-AUC than models trained to produce free-form or stepwise rationales (Page 3). Specifically, no-rationale achieved a mean ROC-AUC of 0.734, compared with 0.604 for free-rationale and 0.592 for stepwise-rationale (Page 3). This degradation persists across model families and data scales, and is not resolved by using a reasoning-oriented base model (Page 1).

Analysis of Rationale Quality

The failure is not explained by poor rationale quality, as human expert annotation confirmed that the generated rationales are medically accurate and faithfully grounded in patient-specific evidence (Page 1). Furthermore, few-shot experiments showed that when the same style of rationale is used as a demonstration rather than a training target, it improves performance over zero-shot baselines (Page 1). The root cause was identified as a structural conflict between narrative plausibility and discriminative optimization (Page 1).

Structural Conflict Explained

The structural conflict arises because a medically plausible rationale must tell a coherent story about why a patient’s history is consistent with their label, emphasizing broad morbidity markers that read as clinically relevant (Page 1). Discriminative fine-tuning, conversely, requires learning which features distinguish future cases from matched controls in this specific cohort (Page 1). These two objectives diverge when the signal is distributed across many patient-specific combinations of features; training to reproduce plausible rationales redirects optimization budget away from learning the discriminative boundary (Page 1).

Implications for Clinical Prediction

The results suggest that label-only fine-tuning is not a weaker approach because it keeps the optimization target directly aligned with the prediction task (Page 4). The degradation is not a complete loss of risk information, as few-shot experiments demonstrated that rationales carry genuine discriminative signal when used as demonstrations (Page 4). However, generated rationales are unreliable for this setting because their plausibility is at best weakly correlated with their discriminative informativeness (Page 4).

Limitations and Future Directions

This study is bounded by its focus on a single disease and cohort, 7–8B open models, and LLM-generated rather than clinician-written rationales (Page 9). Future work should examine whether labelweighted loss functions, discriminative rationale filtering, or hybrid objectives can recover the benefits of rationale-based training while preserving the discriminative signal that direct label supervision provides (Page 9). The findings caution against using synthetic rationales as a shortcut for trustworthy explanation in high-stakes domains (Page 9).

Conclusion

Rationale-based SFT appears to shift optimization toward reproducing narrative reasoning rather than learning the discriminative boundary between cases and controls in this setting (Page 5). This clarifies when rationale supervision may help, when it may harm, and how it should be used in high-stakes clinical prediction (Page 5). The study concludes that label-only fine-tuning is superior to rationale-based SFT for five-year ADRD prediction on longitudinal health histories (Page 16).

--- Page 1 ---

Supervised fine-tuning with synthetic rationale hurts real world disease prediction.

--- Page 2 ---

No rationale sex: Male medical history: F10 alcohol related disorder: 58.4 G40 epilepsy: 62.3 F32 depressive episode: 69.3 BMI high: 70.0 Fluid intelligence low: 70.0 J06 acute upper respiratory infection: 73.6 I67 cerebrovascular disease: 73.1 E78 lipidaemias: 74.7 ADRD Prediction (0 or 1): 1 Free rationale Same disease history. ADRD This male has multiple mid-to-late life factors that plausibly precede Alzheimer’s or related dementia, including alcohol-related disorder, epilepsy, low fluid intelligence, slowed matching time, cerebrovascular disease, and later dyslipidaemia. The combination of cognitive testing abnormalities and cerebrovascular disease in the early 70s makes an ADRD code within the next five years plausible. Prediction (0 or 1): 1 Stepwise rationale Same disease history. ADRD Step 1: Relevant ADRD-related events include I67 at 73.1, earlier G40 and F32, and cognitive testing abnormalities at 70.0. Step 2: Cerebrovascular disease in the early 70s, together with prior cognitive deficits, makes near-term ADRD risk high.

1. Discriminative Rationale Filtering for Training Systems This system should implement a mechanism to filter synthetic rationales based on their discriminative signal rather than just medical plausibility. This addresses the finding that a plausible rationale must tell a coherent story about why a patient’s history is consistent with the groundtruth label while acknowledging that the generated rationales are medically accurate and faithfully grounded in patient-specific evidence, leading to better performance.

2. Optimization Objective Separation for Fine-Tuning This improvement involves decoupling the objective function during supervised fine-tuning so that it does not conflate narrative plausibility with discriminative optimization. This prevents the optimization from redirect[ing] its budget away from learning the discriminative boundary that actually separates cases from controls when training on generated rationales, as observed in Section 4.3.

3. Inference-Time Demonstration Utilization for Prompt Engineering This system should be designed to leverage generated rationales during inference not as training targets but as demonstrations, similar to few-shot experiments. This is supported by the finding that few-shot experiments show that the same rationales improve performance when used as inferencetime demonstrations rather than training targets.

4. Contextual Risk Signal Weighting for Feature Learning This system should learn to distinguish between broad morbidity markers and features that are actual discriminators in sparse, heterogeneous datasets. This is achieved by focusing the model's learning on features that are disproportionately represented among future ADRD cases relative to matched controls, as shown in Figure 4A and C.

Supervised fine-tuning with synthetic rationales is widely assumed to improve clinical prediction by teaching models not only what to predict but why. We test this assumption on five-year Alzheimer's disease and related dementias (ADRD) prediction from longitudinal health histories. In this setting, we generate training rationales using the correct labels, then fine-tune models to output a rationale followed by a label. Across a large-scale controlled experiment of 504 configurations, we find that Rationale First SFT consistently and substantially hurts predictive performance relative to Label Only fine-tuning. The degradation persists across model families and data scales, and is not resolved by using a reasoning-oriented base model. A fixed-protocol cardiovascular disease (CVD) validation reproduces similar findings. Crucially, the failure is not explained by poor rationale quality: human expert annotation confirms that the generated rationales are medically accurate and faithfully grounded in patient-specific evidence, and few-shot experiments show that the same rationales improve performance when used as inference-time demonstrations rather than training targets. A simple label-ordering intervention clarifies the mechanism: placing the prediction before the same rationale restores predictive performance. This suggests that conditioning predictions on generated rationales contributes to the degradation. We hope our work paves the way toward a more precise understanding of when and how rationale-based supervision helps and when it does not, guiding the responsible development of language models for high-stakes clinical prediction.