'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

arXiv:2606.04906 · cs.CL, cs.AI · Submitted 2026-06-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "'Your AI Text is not Mine'".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper today called "Your AI Text is not Mine: Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions" by Nils Dycke, Marina Sakharova, Nico Daheim, and Iryna Gurevych.

Jane: It sounds like they are tackling the big problem where people don't really know what constitutes harmful or acceptable use of AI-generated text because existing datasets just set their own rules without really looking at how people actually create things.

Lu: Exactly. They argue that the current literature on AI text detection is missing a common understanding of what truly counts as harmful use, so they are defining new ways to think about this problem.

Meng: I'm wondering what kind of real-world needs they’re trying to connect these definitions to, because usually these models just look at text in isolation without thinking about how it actually gets used by people.

Lalam: This paper is proposing a way to make sure the notions they define are actually connected to real-world human and AI co-creation, which is a big deal for making detection more meaningful.

Tom: Right. So the core idea is that we need to stop just using whatever definition an existing dataset throws at us and start defining what we mean by AI text based on how it's actually made.

Jane: It seems they are proposing a systematic way to define these notions across three dimensions: the standard for acceptable use, the assumptions about how the text was made, and how much those assumptions can be violated when someone tries to blend unacceptable AI use into something that looks okay.

The paper's summary: Tom: Let's look at what they actually did in this paper. They collected this new benchmark called AITDNA, which is a dataset of over three hundred fifty texts from ninety-nine different human authors.

Jane: What makes AITDNA interesting is that unlike previous work, it records all the detailed human and AI interactions during the writing process—the prompts and even the text edits.

Lu: That level of detail is what lets them make their definitions notion-agnostic, meaning they aren't locked into just one way of thinking about how a text was created.

Meng: So it’s not just a finished block of text; it’s the messy process, which is something I wish we saw more of when we try to deploy these things in production.

Lalam: They are using this AITDNA benchmark to test various existing AI detection methods and show that they often only work well for specific, very narrow assumptions.

Tom: They also break down the notions into two main groups: genesis-based ones, which ask if a human produced the text based on how it was made, and population-based ones that check if the text looks like general human text.

Jane: The paper shows that these existing datasets often encode unrealistic assumptions about how AI is used in real life, which is where their main critique comes from.

The paper's improvements: Tom: So what are the specific improvements they propose to fix this gap in detection literature?

Jane: They formalize these notions by mapping every single token into a label, either Human, AI, or human-AI-mixed. This lets them define parameters for both the genesis notions and the population notions.

Lu: For the genesis notions, they use a parameter called tau to define how much an assigned AI label can violate its underlying assumptions about creation.

Meng: And for the population ones, they use a reference population P to check if something resembles human text based on that specific group instead of just some general crowd.

Lalam: The big improvement here is that this framework lets us do a controlled comparison of different notions and analyze the assumptions in prior datasets without them getting in the way.

Tom: And they find that because existing datasets don't represent natural human-AI co-creation, each notion is actually designed for a specific use case where its underlying assumptions are reasonable.

Jane: They conclude that there isn't one detector that performs better than all others across every dataset and notion, which is a pretty realistic assessment of the situation.

Conclusion: Tom: So to wrap up, the main thing they're pushing is that we need these new definitions—the "Your AI Text is not Mine" framework—to guide us in choosing the right detection tool for any specific job.

Jane: They introduce AITDNA as a way to compare different notions and analyze those underlying assumptions in older research, which should help make future work much more consistent.

Lu: It really shifts the focus from just building detectors to understanding exactly what kind of text creation process we are actually measuring when we look for AI signals.

Meng: For deployment, this means that if we use a certain notion to filter text out of production, we need to know exactly what assumptions are being made about the input data before we scale it up.

Lalam: The overall finding is that aligning detectors with these different target labels requires empirical study because the current alignment between them and these different notions isn't solid yet.

Tom: So we end up with this clear picture: by using AITDNA and these formal notions, we can finally start to evaluate AI text detection in a way that actually makes sense for real-world human-AI interaction.

Ubiquitous Knowledge Processing Lab (UKP Lab) · National Research Center for Applied Cybersecurity ATHENE · Zuse School ELIZA

cs.CL, cs.AI

Submitted: 2026-06-03

Updated: 2026-10-19

Code: https://github.com/UKPLab/arxiv2026-aitdna2https:

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: The gist: This research systematically defines various notions of AI-generated text and their characteristics to address the gap in existing detection literature where datasets and approaches often

Key concepts

Genesis-based Notions: These notions define what makes text AI by looking at how it was created. They focus on whether a span was generated entirely by a human or an AI, or if there is a fixed pattern of alternating passages between the two creators. They set standards based purely on the origin of the text.
Population-based Notions: These notions assess text by comparing it to a reference group. The most common type checks if a piece of text resembles general human writing, often using a large corpus. Another type focuses specifically on whether the text matches the style or authorship of one specific human author.
AITDNA Benchmark: AITDNA is a new dataset containing over 350 texts created through real-world human-AI interactions, including prompts and edits from both parties. This dataset is 'notion-agnostic,' meaning it doesn't predefine how text should be labeled, allowing researchers to test different detection theories against natural co-creation examples.
Notion Parameterization: The paper formalizes the notions by using parameters like τ for genetic notions and P for population-based ones. These parameters quantify the degree to which a segment's assigned AI label might violate its underlying genesis assumption, allowing for controlled comparison of different detection theories.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes

Summary

The gist: This research systematically defines various notions of AI-generated text and their characteristics to address the gap in existing detection literature where datasets and approaches often define their own criteria without relating them to real-world needs.

Redefining AITD Notions

The paper analyzes AITD notions across three dimensions: (1) the normative standard that distinguishes acceptable from unacceptable AI use, (2) the genesis assumptions that impose constraints on a text’s genesis to simplify detection, and (3) the attacker model indicating the degree to which these genesis assumptions are violated as an approach to blend unacceptable into acceptable AI use <ref:2606.04906#pg2> The notions fall into two groups: genesis-based (or genetic) notions, which define normative standards based on text genesis and ask did a human produce this span?, and population-based notions, relying on a reference population and ask does this span resemble human text? <ref:2606.04906#pg4>

AITDNA Benchmark

To test existing AITD evaluation practices, the authors collected AITDNA, a novel dataset of more than 350 texts from 99 human authors while recording detailed human-AI interactions during writing <ref:2606.04906#pg3> Unlike prior work, AITDNA includes the detailed prompts and text edits by the humans and the LLM under a realistic writing setting <ref:2606.04906#pg6> The dataset is notionagnostic and aligned with real-world human–AI co-creation <ref:2606.04906#pg8>

Genesis-based Notions

Genetic notions are most prominent, differing in genesis assumption granularity and thus, detection targets <ref:2606.04906#pg9> Specifically, document-level AITD assumes en-bloc generation by a human or an AI, while boundary-level AITD assumes a fixed number of alternating passages between human and AI These notions define normative standards independently of the content or intent of AI passages

Population-based Notions

The dominant population-based notion is membership-based AITD, which relies on a reference corpus and asks does this span resemble human text? The authors propose authorship-identification-based AITD, which uses a reference corpus specific to a given human author rather than a general human population

Standardization and Evaluation

The paper formalizes notions by mapping each token i in d to a label l ∈ Ygen = H, AI, MX according to underlying functions B and M Genetic notions are parameterized by τ, defining the degree to which an assigned AI label of a segment may violate its underlying genesis Population-based notions are parameterized by a reference population P The study finds that existing AITD datasets do not represent natural human-AI cocreation and encode unrealistic assumptions

Key Findings on Performance

The authors find that existing AITD datasets do not represent natural human-AI cocreation and encode unrealistic assumptions Furthermore, they observe that existing AITD datasets do not represent natural human-AI cocreation and encode unrealistic assumptions They conclude that each notion is designed for a specific use-case in which its assumptions are reasonable The study also shows that there is no detector that consistently outperforms all others across datasets and notions

Conclusion

The paper concludes by stating that we introduce AITDNA, a novel, notion-agnostic, dataset consisting of human-AI texts that were co-created in a real-world setting This dataset enables controlled comparison of notions, cross-notion evaluation, and analysis of assumptions in prior datasets The overall finding is that the alignment between detectors and these different target labels warrants empirical study

The paper systematically defines various notions of AI-generated text and their characteristics to address the gap in existing detection literature where datasets and approaches often define their own criteria without relating them to real-world needs.

How it works

The authors systematically define various notions of AI-generated text and their characteristics They analyze AITD notions along three dimensions: (1) the normative standard that distinguishes acceptable from unacceptable AI use, (2) the genesis assumptions that impose constraints on a text’s genesis to simplify detection, and (3) the attacker model indicating the degree to which these genesis assumptions are violated as an approach to blend unacceptable into acceptable AI use

AITDNA Benchmark

The authors collected AITDNA, a novel dataset of more than 350 texts from 99 human authors while recording detailed human-AI interactions during writing Unlike prior work, AITDNA includes the detailed prompts and text edits by the humans and the LLM under a realistic writing setting

Genesis-based Notions

Genetic notions are most prominent, differing in genesis assumption granularity and thus, detection targets Specifically, document-level AITD assumes en-bloc generation by a human or an AI, while boundary-level AITD assumes a fixed number of alternating passages between human and AI These notions define normative standards independently of the content or intent of AI passages

Standardization and Evaluation

The paper formalizes notions by mapping each token i in d to a label l ∈ Ygen = H, AI, MX according to underlying functions B and M Genetic notions are parameterized by τ, defining the degree to which an assigned AI label of a segment may violate its underlying genesis Population-based notions are parameterized by a reference population P The study finds that each notion is designed for a specific use-case in which its assumptions are reasonable

Key Findings on Performance

The authors find that existing AITD datasets do not represent natural human-AI cocreation and encode unrealistic assumptions Furthermore, they observe that there is no detector that consistently outperforms all others across datasets and notions

Improvements for AI systems

Bold header: Explicit Notion Specification for AITD

This improvement involves moving beyond simplistic notions by utilizing notions of AITD which are formally defined as a mapping from tokens to labels based on a segmentation algorithm and a labeling function, such as Definition 2 (Notion). This allows researchers to precisely define the normative standard, genesis assumptions, and attacker model for detection.

Bold header: Dataset Neutrality via AITDNA

The creation of AITDNA ensures that datasets are notion-agnostic and aligned with real-world human–AI co-creation, enabling a controlled comparison of notions, cross-notion evaluation, and analysis of assumptions in prior datasets. This allows for testing whether existing detectors perform well across different notion assumptions.

Bold header: Empirical Evaluation of Notion Impact

The system can perform empirical studies by testing detectors under varying notions chosen to reflect plausible real-world needs, specifically showing what impact do notions and their parameters have on evaluation. This directly addresses the question: what impact do notions and their parameters have on evaluation?

Abstract

Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.