Provable Training Data Identification for Large Language Models

arXiv:2510.09717 · cs.LG, cs.AI · Submitted 2025-10-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Provable Training Data Identification for Large Language Models".

Jane: Here's the gist of the paper. The gist: This work formalizes training data identification as a set-level inference problem and proposes Provable Training Data Identification (PTDI), a distribution-free method that enables provable and strict false identification rate control.

Tom: Problem Formulation The paper shifts the focus from instance-wise classification to set-level inference, enabling provable error control To address the limitation, we shift our focus from instancewise decisions to set-level inference In this work, we formally define the problem of provable training data identification, where the objective is to construct a selection set from the test data that contains a provable proportion of true member samples Suppose we have access to a target model θ, a calibration set Dcal of size n and a test set Dtest = The paper shifts our focus from instancewise decisions to set-level inference Here, Mi ∈

0, 1: denotes the true membership label, where Mi = 1 indicates that Xi was used to train θ Then we formulate the problem within the framework of multiple hypothesis testing Our goal is to select a subset of indices S ⊆ The paper shifts our focus from instancewise decisions to set-level inference from Dtest such that the false identification rate (FIR) is controlled at a user-specified level α ∈ (0, 1) FIR = E"Pmj=1 1if Mn+j = 0, j ∈ S max(S, 1) <= α where the quantity inside the expectation is the false identification proportion (FIP), representing the fraction of false identifications in the selected set S At this guarantee, we also desire S containing true training data points as much as possible, which is quantified by power: Power = E" Pmj=1 1if j ∈ S, Mn+j = 1 max(1, Pmj=1) > Provable Training Data Identification (PTDI) The proposed method computes detection scores for each data point and then constructs valid p-values via conformal inference To improve power, we further estimate the training-data proportion in the test set via a Jackknife-corrected Beta boundary (JKBB) estimator, and use it to rescale the p-values before applying the Benjamini–Hochberg procedure By applying the Benjamini–Hochberg (BH) procedure to the scaled p-values, we select a subset of data points with provable and strict false identification control This method is compatible with both blackbox outputs and white-box gradient-based scores, and is complementary to existing detection methods, making it a practical tool for training data identification Estimate Data Usage Proportion The paper details the Jackknife-corrected Beta Boundary (JKBB) estimator as the required data usage proportion estimator E needed by Algorithm 1 To estimate the data usage proportion, we rely on accurately estimating the boundary density ftest(1) Let ftest(p) denote the marginal density of these p-values across the entire test set Since each test point is either a non-member (Mn+j = 0) or a member (Mn+j = 1), ftest admits a two-component mixture decomposition ftest(p) = (1−πtest) f(p M = 0) + πtest f(p M = 1) A key observation is that under the null hypothesis, conformal p-values are approximately uniform on

0, 1: , hence f(p M = 0) = 1 Therefore, estimating πtest reduces to estimating the p-value density at the boundary p = 1 Theoretical Guarantees and Scaling Procedure The method establishes theoretical guarantees for PTDI by showing consistency of the JKBB estimator at the boundary, which supports the validity of the scaled p-value procedure and ensures FIR control The asymptotic FIR control of the plug-in scaled BH procedure satisfies lim sup m→∞ FIRm ⩽ 1 − πtest / π0,∞ α This demonstrates that our method effectively controls FIR across all tested ρ values Empirical Validation and Power Improvement Extensive experiments across diverse settings demonstrate that PTDI achieves higher power than prior methods while strictly controlling the FIR For instance, on WikiMIA with Pythia-1.4B at a target FIR of 0.05, our method achieves an empirical FIR of 0.0494, whereas the approach of Hu et al.

Jane: (2025) yields 0.1311 We further demonstrate that our p-value scaling procedure improves power over the vanilla conformal approach Specifically, on WikiMIA with GPT-NeoX-20B at a target FIR of 0.5, our method improves power from 0.5177 to 0.8618 using the MIN-K% detection score In this paper, we formalize training-data identification as a set-level inference problem and introduce PTDI, a method that provides rigorous statistical guarantees for the identified set Our method leverages conformal p-values and the BH procedure to obtain distribution-free guarantees using only non-training calibration data To improve power, we introduce a novel JKBB estimator for the data usage proportion Leveraging this estimator, we apply a p-value scaling strategy that enhances power while maintaining theoretical guarantees Extensive experiments show that PTDI consistently outperforms existing methods in power while maintaining strict FIR control across diverse detection scores Our method enables reliable training data identification in high-stakes applications, such as data copyright litigation, where strict false identification control is required Limitation and Further Work Though our method provides rigorous theoretical guarantees to control the false identification, it requires a calibration set of unseen data that is distributionally similar to the test set A substantial divergence between the calibration and test data may compromise the validity of these guarantees To address this issue, handling distribution shift by weighted conformal p-values may be a promising direction for future work Discussion on Instance-wise Inference The distinction between average FPR and FIR is critical in practice, as the average FPR conflates the overall error rate with the per-hypothesis Type I error rate This is precisely the quantity measured by the false identification rate (FIR), the statistically sound tool for this task Related Work on Conformal Inference Conformal inference provides distribution-free uncertainty quantification by exploiting exchangeability A central object in this framework is the conformal p-value, which compares the nonconformity score of a test point with those of calibration samples and is valid under the null hypothesis without requiring parametric assumptions By contrast, a recent line of work uses conformal p-values for selection to cast training data identification as a multiple-testing problem and control the false identification rate of the selected set Related Work on Estimating Data Usage Proportion Our proposed JKBB estimator is significantly more practical, as it only requires access to the target model, a set of confirmed non-member data, and the test set This problem was recently formalized as Dataset Usage Cardinality Inference (DUCI) by Tong et al.

Tom: (2025) However, this approach requires training reference models to estimate the necessary statistics, rendering it unsuitable for pre-training data detection in large-scale models where the training process is opaque and prohibitively expensive Related Work on Membership Inference Attacks In this paper, we propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods We propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods This work builds on the distribution-free conformal inference framework (Vovk et al., 2003; 2005; Bates et al., 2023; Jin & Candes’, 2023), allowing us to achieve rigorous FIR control without relying on strong distributional assumptions We propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods Additional Related Work on False Identification Rate Control A standard approach for controlling such set-level error is the Benjamini–Hochberg (BH) procedure, but it is conservative since it assumes that all null hypotheses might be true (number of true nulls m0 = m) To address this, a variety of adaptive methods including BH-Storey (Storey, 2002), BKY (Benjamini et al.

Paper summary: Tom: We've talked about the basic idea of this paper now, focusing on the shift from instance-wise to set-level inference and the PTDI approach. What does that actually claim?

Jane: They formally define provable training data identification as a problem where you construct a selection set from the test data that contains a provable proportion of true member samples.

Lu: So, they start by defining the true membership label M i, where M i = one means the sample was used to train the model theta <ref:2510.09717#pg1>. Then they formulate it as multiple hypothesis testing with hypotheses H 0,j that a sample is not training data <ref:2510.09717#pg2>.

Meng: That sounds like heavy math. What's the main goal of this formulation in terms of what they are trying to achieve?

Lalam: The goal is to select a subset of indices S from the test set such that the false identification rate, FIR, is controlled at a user-specified level alpha.

Tom: So, they are aiming for a guarantee on the error rate of the entire selected set rather than just hoping individual predictions are right.

Jane: And to achieve this guarantee, they propose PTDI as a distribution-free approach that enables provable and strict false identification rate control.

Lu: Specifically, their method computes conformal p-values for each data point using a set of known unseen data and then develops a novel Jackknife-corrected Beta boundary estimator to estimate the training data proportion of the test set.

Meng: That estimation step is crucial because it allows them to scale these p-values, which is what lets them apply the Benjamini–Hochberg procedure later on.

Lalam: By applying the Benjamini–Hochberg procedure to these scaled p-values, they select a subset of data points with provable and strict false identification control.

Tom: So, to summarize this paper "Provable Training Data Identification for Large Language Models", they formalize the problem as set-level inference and propose PTDI for provable error control.

Jane: And the importance is that it provides a distribution-free framework that lets you get a guarantee on the false identification rate by using only non-training calibration data.

Lu: This means we don't have to assume too much about the underlying distribution of our test set for the guarantees to hold.

Meng: If they can provide that, it makes it much more trustworthy when we try to use these detection methods on real-world model outputs.

Lalam: It also means this method is compatible with both blackbox outputs and whitebox gradient-based scores, making it a practical tool.

Conclusion: Tom: So, wrapping up the discussion on "Provable Training Data Identification for Large Language Models", what's the big picture implication of this work?

Jane: The authors introduce PTDI as a method that provides rigorous statistical guarantees for the identified set, which is what really sets it apart.

Lu: They leverage conformal p-values and the Benjamini–Hochberg procedure to obtain distribution-free guarantees using only non-training calibration data.

Meng: And they improve power by introducing a novel Jackknife-corrected Beta boundary estimator for the data usage proportion, which lets them rescale those p-values.

Lalam: This scaling strategy enhances power while maintaining theoretical guarantees across all tested rho values.

Tom: So, the main thing is that this method consistently outperforms prior methods in power while strictly controlling the false identification rate across diverse detection scores.

Jane: For someone listening who doesn't read the paper, it means we have a more reliable way to perform training data identification in high-stakes applications where strict false identification control is required.

Lu: This method enables reliable training data identification in high-stakes applications, such as data copyright litigation, where strict false identification control is required.

Meng: It gives us a method that provides rigorous statistical guarantees for the identified set, which is what really sets it apart from previous work.

Lalam: The paper's versatility means it can be integrated with most MIA methods to ensure strict FIR control without relying on strong distributional assumptions.

Zhenlong Liu, Hao Zeng, Weiran Huang, Hongxin Wei

Southern University of Science and Technology

cs.LG, cs.AI

Submitted: 2025-10-10

Updated: 2026-08-07

Journal ref: Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), PMLR 306:79648-79674

Code: https://github.com/zhenlong-liu/Provable_Training_

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: The gist: This work formalizes training data identification as a set-level inference problem and proposes Provable Training Data Identification (PTDI), a distribution-free method that enables

Key concepts

Set-level Inference: Instead of checking each data point individually, this method treats the identification task as a problem concerning a whole set of test data. The goal is to select a subset from this set that contains a provable proportion of actual training data points, offering stronger statistical guarantees than looking at one sample at a time.
Conformal p-values: These are scores calculated for each test point by comparing its nonconformity score against calibration samples. They are powerful because they provide distribution-free uncertainty quantification, meaning they work well even if the underlying data distributions are unknown or complex, without needing strong parametric assumptions.
Jackknife-corrected Beta boundary (JKBB) estimator: This is a new statistical tool used to estimate the true proportion of training data within the test set. It helps scale the conformal p-values accurately, improving the method's power to find training data while maintaining strict control over false identifications.

Terminology

Provable Training Data Identification — 2× in this paper

Summary

The gist: This work formalizes training data identification as a set-level inference problem and proposes Provable Training Data Identification (PTDI), a distribution-free method that enables provable and strict false identification rate control.

Problem Formulation

The paper shifts the focus from instance-wise classification to set-level inference, enabling provable error control To address the limitation, we shift our focus from instancewise decisions to set-level inference In this work, we formally define the problem of provable training data identification, where the objective is to construct a selection set from the test data that contains a provable proportion of true member samples Suppose we have access to a target model θ, a calibration set Dcal of size n and a test set Dtest = The paper shifts our focus from instancewise decisions to set-level inference Here, Mi ∈ [0, 1] denotes the true membership label, where Mi = 1 indicates that Xi was used to train θ Then we formulate the problem within the framework of multiple hypothesis testing Our goal is to select a subset of indices S ⊆ The paper shifts our focus from instancewise decisions to set-level inference from Dtest such that the false identification rate (FIR) is controlled at a user-specified level α ∈ (0, 1) FIR = E"Pmj=1 1if Mn+j = 0, j ∈ S max(S, 1) <= α where the quantity inside the expectation is the false identification proportion (FIP), representing the fraction of false identifications in the selected set S At this guarantee, we also desire S containing true training data points as much as possible, which is quantified by power: Power = E" Pmj=1 1if j ∈ S, Mn+j = 1 max(1, Pmj=1) >

Provable Training Data Identification (PTDI)

The proposed method computes detection scores for each data point and then constructs valid p-values via conformal inference To improve power, we further estimate the training-data proportion in the test set via a Jackknife-corrected Beta boundary (JKBB) estimator, and use it to rescale the p-values before applying the Benjamini–Hochberg procedure By applying the Benjamini–Hochberg (BH) procedure to the scaled p-values, we select a subset of data points with provable and strict false identification control This method is compatible with both blackbox outputs and white-box gradient-based scores, and is complementary to existing detection methods, making it a practical tool for training data identification

Estimate Data Usage Proportion

The paper details the Jackknife-corrected Beta Boundary (JKBB) estimator as the required data usage proportion estimator E needed by Algorithm 1 To estimate the data usage proportion, we rely on accurately estimating the boundary density ftest(1) Let ftest(p) denote the marginal density of these p-values across the entire test set Since each test point is either a non-member (Mn+j = 0) or a member (Mn+j = 1), ftest admits a two-component mixture decomposition ftest(p) = (1−πtest) f(p M = 0) + πtest f(p M = 1) A key observation is that under the null hypothesis, conformal p-values are approximately uniform on [0, 1], hence f(p M = 0) = 1 Therefore, estimating πtest reduces to estimating the p-value density at the boundary p = 1

Theoretical Guarantees and Scaling Procedure

The method establishes theoretical guarantees for PTDI by showing consistency of the JKBB estimator at the boundary, which supports the validity of the scaled p-value procedure and ensures FIR control The asymptotic FIR control of the plug-in scaled BH procedure satisfies lim sup m→∞ FIRm ⩽ 1 − πtest / π0,∞ α This demonstrates that our method effectively controls FIR across all tested ρ values

Empirical Validation and Power Improvement

Extensive experiments across diverse settings demonstrate that PTDI achieves higher power than prior methods while strictly controlling the FIR For instance, on WikiMIA with Pythia-1.4B at a target FIR of 0.05, our method achieves an empirical FIR of 0.0494, whereas the approach of Hu et al. (2025) yields 0.1311 We further demonstrate that our p-value scaling procedure improves power over the vanilla conformal approach Specifically, on WikiMIA with GPT-NeoX-20B at a target FIR of 0.5, our method improves power from 0.5177 to 0.8618 using the MIN-K% detection score

Conclusion

In this paper, we formalize training-data identification as a set-level inference problem and introduce PTDI, a method that provides rigorous statistical guarantees for the identified set Our method leverages conformal p-values and the BH procedure to obtain distribution-free guarantees using only non-training calibration data To improve power, we introduce a novel JKBB estimator for the data usage proportion Leveraging this estimator, we apply a p-value scaling strategy that enhances power while maintaining theoretical guarantees Extensive experiments show that PTDI consistently outperforms existing methods in power while maintaining strict FIR control across diverse detection scores Our method enables reliable training data identification in high-stakes applications, such as data copyright litigation, where strict false identification control is required

Limitation and Further Work

Though our method provides rigorous theoretical guarantees to control the false identification, it requires a calibration set of unseen data that is distributionally similar to the test set A substantial divergence between the calibration and test data may compromise the validity of these guarantees To address this issue, handling distribution shift by weighted conformal p-values may be a promising direction for future work

Discussion on Instance-wise Inference

The distinction between average FPR and FIR is critical in practice, as the average FPR conflates the overall error rate with the per-hypothesis Type I error rate This is precisely the quantity measured by the false identification rate (FIR), the statistically sound tool for this task

Related Work on Conformal Inference

Conformal inference provides distribution-free uncertainty quantification by exploiting exchangeability A central object in this framework is the conformal p-value, which compares the nonconformity score of a test point with those of calibration samples and is valid under the null hypothesis without requiring parametric assumptions By contrast, a recent line of work uses conformal p-values for selection to cast training data identification as a multiple-testing problem and control the false identification rate of the selected set

Related Work on Estimating Data Usage Proportion

Our proposed JKBB estimator is significantly more practical, as it only requires access to the target model, a set of confirmed non-member data, and the test set This problem was recently formalized as Dataset Usage Cardinality Inference (DUCI) by Tong et al. (2025) However, this approach requires training reference models to estimate the necessary statistics, rendering it unsuitable for pre-training data detection in large-scale models where the training process is opaque and prohibitively expensive

Related Work on Membership Inference Attacks

In this paper, we propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods We propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods This work builds on the distribution-free conformal inference framework (Vovk et al., 2003; 2005; Bates et al., 2023; Jin & Candes’, 2023), allowing us to achieve rigorous FIR control without relying on strong distributional assumptions We propose a versatile method that ensures strict FIR control and can be integrated with most MIA methods

Additional Related Work on False Identification Rate Control

A standard approach for controlling such set-level error is the Benjamini–Hochberg (BH) procedure, but it is conservative since it assumes that all null hypotheses might be true (number of true nulls m0 = m) To address this, a variety of adaptive methods including BH-Storey (Storey, 2002), BKY (Benjamini et al.

Improvements for AI systems

Proof of Training Data Identification (PTDI) can rigorously control false identification rate by using conformal p-values and a novel Jackknife-corrected Beta boundary estimator to estimate data usage proportion, which allows for provable and strict false identification rate control.
AI systems can identify specific subsets of training data with provable statistical guarantees, such as in copyright litigation or privacy auditing, by ensuring the False Identification Rate (FIR) is strictly controlled at a user-specified level alpha.
The system can achieve higher power in identifying true training data members compared to prior methods by applying a p-value scaling procedure that adjusts for the estimated proportion of training data in the test set.
The system can provide an adjusted moment estimator to further improve identification power when a small set of confirmed training data members is available, ensuring consistency with strict false identification control.
The system can be used as a distribution-free tool for training data identification, compatible with both blackbox outputs and white-box gradient-based scores, making it practical for various model types like LLMs and VLMs.