Hidden Data Challenges in Clinical Machine Learning

As a fastidious researcher, I must synthesize these disparate excerpts into a coherent, detailed summary of the paper, ensuring all critical findings and methodological rigor are captured without introducing external commentary or speculation.

Here is the comprehensive summary based on the provided text:

The paper addresses the critical issue that widely reused preprocessing pipelines for electronic health records (EHRs) often encode implicit assumptions regarding information availability timing, clinical episode definitions, and missing data handling. These unexamined assumptions can propagate across subsequent studies when pipelines are adapted or reused, leading to subtle but significant alterations in the signals models rely upon.

The core investigation uses MIMIC-IV as a case study to systematically identify and quantify violations of three complementary forms of preprocessing alignment:

1. Temporal Alignment: The congruence between event occurrence and the actual availability of information. Violations manifest as temporal leakage (e.g., random splits allocating multiple admissions from the same patient into training/test sets, or confusion between overlapping concepts like ED stays versus ICU stays) and temporal target misalignment (especially in prediction tasks based on short post-admission windows).

2. Semantic Alignment: The consistency between the evidence contained within a data source and the specific clinical endpoint being targeted. This arises when endpoints conflate distinct phenomena (e.g., in-hospital versus out-of-hospital mortality) or when a single modality is inherently heterogeneous, providing inconsistent evidence for a specific disease endpoint (e.g., chest X-rays not directly mapping to all ICD code categories defining a respiratory disease).

3. Population Alignment: The match between the population from which a feature or modality is observed and the population intended for which the preprocessing is applied. A key concern here involves missingness handling; adopting mean imputation for entirely missing measurements can introduce selection/imputation bias if the empirical mean does not correspond to a clinically normal or unobtrusive value within that specific patient cohort.

These misalignments are operationalized through four failure modes: Data leakage, Temporal target misalignment, Semantic feature–target misalignment, and Selection/Imputation bias. The authors demonstrate that these choices can change the underlying features models rely on, even when aggregate predictive performance metrics (such as AUROC or AUPRC) show only modest differences.

To mitigate these issues and ensure reproducibility, the paper derives practical recommendations for designing and evaluating MIMIC preprocessing pipelines. These recommendations emphasize prioritizing transparency, ensuring temporal validity by explicitly defining the modeling perspective (distinguishing between patient-state/exposure modeling versus clinician-knowledge/decision-support modeling), validating endpoint and feature-task relevance, applying context-aware imputation strategies rather than unjustified exclusion or imputation, and validating multimodal contributions against unimodal baselines.

The paper concludes that preprocessing errors fundamentally alter the features models depend on, thereby changing the logic underlying predictions. While aggregate performance differences are often small (less than 0.5% difference in AUROC), this does not negate the importance of careful preprocessing. The authors stress that storetime should be interpreted as a proxy for information availability, not a direct record of observation time, and that clinically informed imputation values are heuristics requiring adaptation to different populations and tasks.

For rigorous verification and auditing, the paper details extensive documentation requirements:

• Model & Modality Evaluation: Multimodal contributions are verified by comparing unimodal baselines on identical evaluation episodes and data partitions, using confidence intervals to account for patient-level clustering, and statistically examining endpoint subgroups with clinical input.

• Pipeline Auditing & Reproducibility: Comprehensive reporting is mandated at every step of the pipeline. This includes reporting cohort counts and outcome prevalence after every inclusion/exclusion/join step, detailing attrition (e.g., join attrition, duplicate amplification), and documenting all criteria used for variable selection, aggregation rules, unit harmonization, outlier handling, modeling perspective anchors (patient-state vs. clinician-knowledge), and prediction anchors. Furthermore, source tables, timestamp columns with their availability rules (storetime interpretation), workflow stages, imputation strategies (including justification for clinical heuristics), modality-coverage rules, and exact software versions/random seeds must be provided. Preprocessing failure cases are reported quantitatively or via representative examples.

In essence, the paper provides a rigorous framework—a cohesive list of steps and questions—to prevent these pitfalls by demanding exhaustive documentation and validation across the entire preprocessing lifecycle, acknowledging that while this checklist is not exhaustive, ongoing vigilance remains essential for dataset-specific clinical and technical validation.

1. Improve preprocessing pipelines by explicitly documenting all decisions to support reproducibility, as recommended by Prioritizing transparency: Document all variable selection criteria, general preprocessing decisions, timestamp choices, outlier handling procedures, and imputation strategies to support reproducibility. This allows researchers to audit assumptions that may propagate across subsequent studies when preprocessing pipelines are reused or adapted.

2. Implement availability-aware timestamp selection based on the modeling perspective. For instance, for clinician-knowledge or decision-support modeling, features should be restricted to information that could reasonably have been available by the prediction time according to the available workflow and timestamp proxies.

3. Refine temporal alignment by choosing timestamps that match the intended representation. When using patient-state or exposure modeling, event timestamps such as charttime, specimen collection time, or starttime are defensible, whereas for clinician-knowledge or decision-support modeling, the choice should be conservative and use the later timestamp.

4. Address temporal ambiguity in tabular data by distinguishing between event occurrence and information availability. For laboratory data, features should satisfy both charttime ≤ tpred and storetime ≤ tpred when storetime is available as a proxy for result availability for decision-support models, or anchor observations to specimen time for patient-state models.

5. Mitigate selection/imputation bias by avoiding the transfer of statistics from subgroups to the broader cohort unless representativeness can be established. The paper advises that Imputing values from the statistical properties of an observed subgroup onto the broader cohort is inadvisable because the observed subgroup generally cannot be assumed to represent the full population.

6. Enhance multimodal model validation by comparing models against unimodal baselines and modality-ablation baselines. This helps determine if high aggregate image feature importance does not necessarily imply that the imaging modality improves prediction or contributes endpoint-relevant signal, as shown in Figure 8.

7. Utilize SHAP values to provide complementary information about model reliance rather than interpreting them as definitive evidence of clinical relevance. The paper cautions that SHAP values characterize model attribution but do not establish causal or clinically meaningful modality contribution.

Widely reused preprocessing pipelines for electronic health records can encode assumptions about when information became available, what clinical episode a label represents, and what missingness means. These assumptions are rarely audited systematically and may propagate across subsequent studies when preprocessing pipelines are reused or adapted. Using MIMIC-IV as a case study, we identify and quantify violations of three complementary forms of preprocessing alignment, i.e., temporal, semantic, and population alignment, which manifest as temporal leakage, feature-task misalignment, and biased imputation. We show how ambiguities in timestamp semantics, episode definitions, endpoint construction, modality selection, and missingness handling can alter the information available to a model and affect the interpretation of its predictions. Through analyses of MIMIC-IV data and empirical case studies, we demonstrate that such choices may remain largely invisible in aggregate predictive performance while changing the signals on which models rely. We derive practical recommendations for temporally valid, clinically aligned, and reproducible preprocessing of MIMIC-IV and related electronic health record datasets. The code for our experiments, including our proposed preprocessing, is available at: https://github.com/m2l-lab/mimic-iv-preprocess.