Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests

Detailed Research Summary: Can Synthetic Data Reproduce Real-World Findings in Epidemiology?

This research investigates a critical, underexplored question in synthetic data generation for epidemiology: Can synthetic data reliably reproduce key findings from real-world epidemiological research while simultaneously preserving patient privacy? The study addresses the limitations of current methods, which are often plagued by poor quality, high computational demands, and complexity for non-experts. Furthermore, existing evaluation strategies frequently fail to adequately measure both statistical utility and the nuances of privacy risks inherent in this domain.

Core Methodology: Adversarial Random Forests (ARF)

The central contribution of this paper is the proposal and rigorous validation of Adversarial Random Forests (ARF) as an efficient and convenient method for synthesizing tabular epidemiological data. The study's strength lies in its application-oriented validation across multiple, diverse real-world datasets and analysis types.

Validation Scope:

To rigorously test ARF's fidelity, the authors replicated statistical analyses from six distinct epidemiological publications. These studies spanned a wide range of medical domains, including:

• Blood pressure

• Anthropometry

• Myocardial infarction (heart disease)

• Accelerometry (activity tracking)

• Loneliness and diabetes

The data utilized were sourced from established cohorts, specifically the German National Cohort (NAKO Gesundheitsstudie), the Bremen STEMI Registry U45 Study (BSR-U45), and the Guelph Family Health Study (GFHS).

Evaluation Metrics and Analysis:

The evaluation was comprehensive, focusing on four key pillars: Utility, Privacy Risk, Generalization, and Runtime efficiency.

1. Utility Assessment:

• For descriptive analyses (like regression), performance was quantified using the Mean Absolute Standardised Difference (MASD) and the Mean Confidence Interval Overlap (CIO).

• For distribution analysis, the researchers computed the mean of the Wasserstein distances (WD) between training and synthetic data across all four subgroups.

2. Privacy Risk Assessment:

• A thorough privacy audit was conducted using multiple attack vectors: distance-based attacks, local density attacks, and classifier-based membership inference attacks (MIAs), as well as attribute inference attacks (AIAs).

• The success of MIAs was specifically measured by the True Positive Rate at a False Positive Rate of 0.01 (TPR@ FPR=0.01).

• AIAs were evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC) for categorical targets and MASD for continuous targets.

3. Generalization:

• To assess the model's ability to capture underlying data structure rather than just memorizing training points, researchers measured the multivariate Wasserstein distance (WD) between synthetic data and unseen test data not used during synthesizer training. The results indicated that ARF achieves competitive generalization to unseen data while maintaining a favorable privacy risk profile compared to alternative synthesizers.

4. Runtime Analysis:

• ARF demonstrated significant computational advantages, operating orders of magnitude faster than deep learning-based approaches (e.g., requiring more than ten seconds) and Bayesian network-based approaches (requiring over two minutes). For instance, ARF and synthpop finished training and sampling in under 0.5 seconds for one study, while Berger et al.'s ARF achieved a median runtime of 35.29 seconds.

Key Findings and Conclusions

The research yielded compelling evidence supporting the efficacy of ARF for epidemiological data synthesis:

• Reliable Replication of Findings: Across all six replicated studies, results generated from ARF-synthesized data consistently aligned with the original findings. This consistency held even for datasets exhibiting relatively low sample size-to-dimensionality ratios, where replication outcomes closely matched the original descriptive and inferential analyses.

• Superior Performance in Low Data Regimes: The study found that a higher participants-to-dimensionality ratio generally led to closer resemblance between synthetic and real data. Consequently, the authors recommend restricting the input data to only those variables relevant to subsequent analyses or performing necessary variable derivations prior to synthesis.

• Optimization of Utility vs. Privacy Trade-off: The study identified that the minimum node size in ARF significantly influences this trade-off: smaller values generally improved utility but increased privacy risks. However, the authors suggest a practical recommendation: using slightly larger node sizes (e.g., 5 or 10 instead of the default of 2) may provide a more balanced default setting for privacy-sensitive applications.

• Favorable Balance: ARF was consistently ranked among synthesizers with the highest utility while simultaneously exhibiting lower MIA risks than synthpop, which showed increased risks in specific instances (e.g., Wienbergen et al., where ARF achieved TPR@ FPR=0.0388 compared to synthpop's 0.100). Overall, ARF demonstrated the most favorable balance between utility and privacy risk when compared against other state-of-the-art synthesizers (including Bayesian Networks and PrivBayes).

• Application Focus: A key strength highlighted is the study’s epidemiology-focused validation. Unlike typical machine learning papers, this work provides application-oriented assessments relevant to epidemiological analyses, explicitly addressing privacy aspects—which are often omitted in standard ML reporting—as non-trivial concerns in this field.

**In summary, the paper concludes that ARF is a practical and valuable tool for synthesizing tabular epidemiological data. It reliably generates high-quality synthetic datasets that closely replicate diverse descriptive and inferential findings across various datasets and analyses, offering a competitive trade-off between utility, privacy preservation, and superior computational efficiency.

1. Adversarial random forests (ARF) can generate high-quality synthetic tabular epidemiological data that reliably reproduce key findings from diverse real-world analyses, as shown by consistently aligning median results with original findings across six studies.

2. The ARF method offers a competitive privacy to utility trade-off, matching or exceeding all alternatives in utility and privacy preservation across four of six datasets while operating orders of magnitude faster than other synthesizers.

3. Researchers can use ARF for task-specific synthesis by restricting the dataset to variables used in subsequent analyses, which further enhanced synthesis quality compared to general synthesis when dataset dimensionality and variable complexity were reduced.

4. The system provides metrics like Mean Absolute Standardized Difference (MASD) and Mean Confidence Interval Overlap (CIO) to quantify deviations between original and synthetic results, allowing for a rigorous assessment of bias in the reproduction of analytical outcomes.

5. The ARF model's performance can be tuned by varying the minimum node size, providing a mechanism to balance utility and privacy risk, suggesting that slightly larger values (e.g., 5 or 10 instead of the default of 2) may therefore provide a more balanced default in privacy-sensitive applications.

6. The ARF system can be extended for more challenging data types by developing a differentially private [43] variant of ARF or an extension for extremely high-dimensional data, such as omics datasets, addressing future research directions.

Synthetic data hold substantial potential to address practical challenges in epidemiology due to restricted data access and privacy concerns. However, many current methods suffer from limited quality, high computational demands, and complexity for non-experts. Furthermore, common evaluation strategies for synthetic data often fail to directly reflect statistical utility and measure privacy risks sufficiently. Against this background, a critical underexplored question is whether synthetic data can reliably reproduce key findings from epidemiological research while preserving privacy. We propose adversarial random forests (ARF) as an efficient and convenient method for synthesizing tabular epidemiological data. To evaluate its performance, we replicated statistical analyses from six epidemiological publications covering blood pressure, anthropometry, myocardial infarction, accelerometry, loneliness, and diabetes, from the German National Cohort (NAKO Gesundheitsstudie), the Bremen STEMI Registry U45 Study, and the Guelph Family Health Study. We further assessed how dataset dimensionality and variable complexity affect the quality of synthetic data, and contextualized ARF's performance by comparison with commonly used tabular data synthesizers in terms of utility, privacy, generalization, and runtime. Across all replicated studies, results on ARF-generated synthetic data consistently aligned with original findings. Even for datasets with relatively low sample size-to-dimensionality ratios, replication outcomes closely matched the original results across descriptive and inferential analyses. Reduced dimensionality and variable complexity further enhanced synthesis quality. ARF demonstrated favourable performance regarding utility, privacy preservation, and generalization relative to other synthesizers and superior computational efficiency.