In-Context Bias Propagation in LLM-Based Tabular Data Generation

The gist: Statistical biases within in-context examples systematically propagate to LLM-generated tabular data, leading to global statistical distortions and compromising downstream fairness, even when utility metrics remain high.

In-Context Bias Propagation

The paper systematically studies how statistical biases within in-context examples propagate to the distribution of synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions In this paper, we systematically study how statistical biases within in-context examples propagate to the distribution of synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions. The authors argue that the same mechanism that allows LLMs to adapt rapidly to new tasks also makes them susceptible to replicating and amplifying demographic disparities present in the context examples.

Modeling In-Context Sensitivity

The authors model in-context learning as approximate Bayesian inference, which suggests that the resulting predictive distribution behaves like a mixture of task-conditional predictors with weights that depend on the prompt. They introduce a two-component decomposition to study in-context influence on tabular synthesis, defined by DG ≈ (1 − αk) D˜0 + αk ΦM(DP). To quantify bias, they relate the ratio of adversarial examples within the prompt to the bias shift µϕ(DG) − µϕ(D0) observed between the reference distribution D0 and the ICL-generated distribution DG.

Experimental Analysis of Bias Levels

The analysis is structured across three levels of granularity: marginal bias propagation, conditional bias propagation, and intersectional propagation. The authors demonstrate that statistical biases present in ICL examples systematically propagate to the distribution of LLM-generated tabular data, and that LLMs capture and propagate biases in a conditional and intersectional manner. They quantify this propagation via linear sensitivity analysis across multiple model families.

Adversarial Bias Injection

The paper introduces an adversarial scenario where a malicious contributor can inject bias into the synthetic dataset via a subset of in-context examples, ultimately compromising the fairness of downstream classifiers for a targeted and protected subgroup. This targeted manipulation can skew the synthetic distribution and compromise the fairness of downstream classifiers, all while maintaining high utility metrics that mask the attack. The adversary controls a fraction π of the k examples in the prompt and injects πk malicious, feature–aligned records into the prompt, while the remaining (1 − π)k examples are sampled i.i.d. from the real dataset.

Mitigation Strategies

The authors evaluate mitigation strategies based on preprocessing in-context examples, demonstrating that while such interventions can attenuate disparity, the inherent sensitivity of LLMs to adversarial prompts remains a persistent challenge. They test four strategies: correlation-based filtering, Fair-SPD, group balanced, and random subset. The results show that Fair-SPD consistently reduces downstream disparity SPDD relative to the unmitigated baseline while preserving comparable F1 in most settings.

Conclusion on Vulnerability

The findings reveal a new attack surface for LLM-based tabular data generation, showing that this manipulation propagates to downstream classifiers, which often amplify the injected bias. The paper concludes that in-context pre-processing defenses are insufficient to fully neutralize the injected bias, highlighting the need for more robust safeguards in generative data pipelines. The susceptibility to in-context bias is a fundamental property of the learning paradigm itself, rather than a behavior emerging solely from model scale or from prompt engineering. The mechanism of in-context bias propagation can be exploited by an adversary to maliciously inject biases for a targeted subgroup, even when the target lies at the intersection of multiple variables. The authors demonstrate that these synthetic biases are captured by downstream models, leading to discriminatory predictions on real test data. The paper shows that fairness can deteriorate substantially while conventional utility metrics remain largely unchanged.

The work reveals a critical vulnerability inherent to the paradigm of using Large Language Models for synthetic tabular data generation, where statistical biases in in-context examples systematically propagate to synthetic distributions and subsequently compromise downstream fairness without degrading standard utility diagnostics. This vulnerability is exploitable by malicious actors who can inject feature-aligned examples to manipulate the output distribution for a targeted subgroup, demonstrating that even small fractions of in-context examples can significantly alter the group-wise label distribution of the synthetic data while preserving macro F1 scores. The research underscores that increasing context window size to improve synthetic data utility inadvertently heightens the risk of replicating historical injustices or adversarial injections, as larger windows faithfully replicate and amplify demographic skews when in-context examples are demographically skewed. Despite evaluating various pre-processing techniques such as correlation-based filtering and Fair-SPD selection, these defenses are insufficient to fully neutralize the injected bias because biases often persist due to the model’s intrinsic priors and the leakage of higher-order correlations not captured by marginal constraints. Ultimately, this research establishes adversarial in-context bias injection as a new vulnerability scheme in LLM-based generation pipelines that affects fairness in synthetic data while maintaining high utility, necessitating more robust safeguards against prompt-level manipulation. The paper shows that these synthetic biases are captured by downstream models, leading to discriminatory predictions on real test data across various architectures and classifiers.

--- Page 1 ---

In-Context Bias Propagation in LLM-Based Tabular Data Generation

Pol G. Recasensa, Alberto Gutierreza, Jordi Torresa, Josep Ll. Berralb, Javier Carnerero-Canoc, Anisa Halimic, Kieran Fraserc aBarcelona Supercomputing Center (BSC), Spain bUniversitat Politècnica de Catalunya (UPC), Spain cIBM Research Ireland Abstract

Large Language Models (LLMs) are increasingly used for synthetic tabular data generation through in-context learning (ICL), offering a practical solution for data augmentation in data scarce scenarios. While prior work has shown the potential of LLMs to improve downstream task performance through augmenting underrepresented groups, these benefits often assume access to a subset of unbiased in-context examples, representative of the real dataset. In real-world settings, however, data is frequently noisy and demographically skewed. In this paper, we systematically study how statistical biases within in-context examples propagate to the distribution of synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions. We further introduce an adversarial scenario where a malicious contributor can inject bias into the synthetic dataset via a subset of in-context examples, ultimately compromising the fairness of downstream classifiers for a targeted and protected subgroup. Finally, we evaluate mitigation strategies based on preprocessing in-context examples, demonstrating that while such interventions can attenuate disparity, the inherent sensitivity of LLMs to adversarial prompts remains a persistent challenge. Our findings highlight a critical new vulnerability in LLM-based data generation pipelines within sensitive domains. Keywords: in-context learning, bias mitigation, synthetic data generation, large language models

--- Page 2 ---

1. Introduction

The scarcity of high-quality, labeled data remains a primary bottleneck in deploying machine learning systems in sensitive domains such as healthcare, finance, and criminal justice. Due to strict privacy regulations and the high cost of manual annotation, practitioners increasingly rely on synthetic data augmentation to build robust classifiers [1]. While traditional generative models [2, 3] have been the standard for tabular synthesis, they often require substantial in-domain data to generate reliable data. Recently, Large Language Models (LLMs) have emerged as a powerful alternative, demonstrating remarkable generalization capabilities through in-context learning (ICL). By conditioning on a small set of ICL examples, LLMs can generate realistic, statistically coherent tabular samples without the need for finetuning [1]. This paradigm leverages the model’s strong semantic priors to augment underrepresented groups, offering a resourceefficient solution for data-scarce regimes. Recent studies show that LLM-generated synthetic data can significantly boost downstream task performance, matching or even surpassing traditional data augmentation methods in data-scarce regimes [4, 5]. However, the efficacy of ICL relies on a critical and often unexamined assumption, which is that in-context examples provided in the prompt are unbiased and independently drawn from the real data distribution. In real-world deployment scenarios –such as Retrieval-Augmented Generation (RAG) pipelines or collaborative data platforms –this assumption rarely holds. In-context examples are frequently noisy, skewed, or dynamically retrieved from unverified sources.

--- Page 3 ---

Clean Examples Background PROMPT Poisoned Examples Clean Examples In-context Examples Background PROMPT

Example 1: Black Example 2: White Example 3: White Example 79: Black Example 80: Black LLM-based Tabular Data Generation Downstream Model Adversarial Bias Injection Bias Propagation Race Attrs Label Utility evaluation Macro F1 Score Fairness Evaluation SPD High disparity Figure 1: End-to-end bias propagation. An adversarial user injects biased examples into the prompt for a targeted group. The LLM replicates these statistical patterns via in-context learning, propagating the bias into the synthetic tabular data, and further compromising downstream fairness.

--- Page 4 ---

Race Attrs Label Utility evaluation Macro F1 Score Fairness Evaluation SPD High disparity Figure 1: End-to-end bias propagation. An adversarial user injects biased examples into the prompt for a targeted group. The LLM replicates these statistical patterns via in-context learning, propagating the bias into the synthetic tabular data, and further compromising downstream fairness.

1. Adversarial bias injection defense can be implemented by applying correlation-based filtering to in-context examples by removing those with high feature magnitudes weighted by correlation scores si = Pj ρj · zi, j. This aims to remove the injected adversarial correlations that steer protected group profiles toward attacker-chosen plausible values.

2. Fair Statistical Parity Difference enforcement can be used as a pre-processing strategy where a selection algorithm iteratively removes the single example contributing most to current disparity until SPD(Scurr) <= epsilon is met. This helps attenuate adversarial bias injection by aligning the prompt examples with a desired level of fairness, although it may not fully eliminate downstream disparities.

3. For complex tasks involving multiple protected attributes, a balanced prompt instruction can be used to generate specific combinations of subgroups, such as one example for each combination of gender and race. This helps disentangle conditional dependence from frequency effects by explicitly instructing the LLM to generate examples that maintain balanced frequency counts across subgroups.

4. Model selection can prioritize larger models or mixture-of-experts architectures when aiming for precise bias propagation control, as they demonstrate greater precision in bias propagation and exhibit stronger conditional propagation compared to smaller models within the same family.

5. Downstream model training should be augmented with a consideration of feature importance analysis, such as Mean Decrease Impurity (MDI), to identify when the downstream classifier begins to rely on the protected attribute even when it is excluded from prediction variables, indicating where bias transfer is occurring.

Large Language Models (LLMs) are increasingly used for synthetic tabular data generation through in-context learning (ICL), offering a practical solution for data augmentation in data scarce scenarios. While prior work has shown the potential of LLMs to improve downstream task performance through augmenting underrepresented groups, these benefits often assume access to a subset of unbiased in-context examples, representative of the real dataset. In real-world settings, however, data is frequently noisy and demographically skewed. In this paper, we systematically study how statistical biases within in-context examples propagate to synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions. We further introduce an adversarial scenario where a malicious contributor can inject bias into the synthetic dataset via a subset of in-context examples, ultimately compromising the fairness of downstream classifiers for a targeted and protected subgroup. Finally, we evaluate prompt-side mitigation strategies based on preprocessing in-context examples and instruction-based defenses, demonstrating that while such techniques can attenuate disparity, the inherent sensitivity of LLMs to adversarial prompts remains a persistent challenge. Our findings highlight a critical new vulnerability in LLM-based data generation pipelines within sensitive domains.