How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions

arXiv:2604.22503 · cs.CL · Submitted 2026-04-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions".

Jane: The gist: Persona distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight,

Tom: First, who's behind it and why it matters.

Title and authors: Lu: They focused on model-level interventions, specifically testing a reranking approach to reduce undesirable distortions, like that polarising effect where AI makes stances seem more extreme.

Tom: They found that reranking successfully reduced this targeted polarising effect by fifty-four point six percent, which was a plus seven point four marginal effect compared to the three point four they saw before at a p value less than zero point zero one.

Meng: But there was a cost to that reduction; writers were significantly less likely to prefer AI-written paragraphs produced under reranking than under no intervention, with preference dropping from fifty-eight point nine percent down to forty-eight point eight percent at a p value less than zero point zero one.

Jane: That suggests that while they can reduce the distortion on the stance aspect, it messes with user acceptance in a way that seems pretty hard to manage.

Lalam: The authors pointed out that these side effects from reranking aren't random; they come from a shared perceptual structure underlying how readers judge things, which is something we have to keep in mind.

Tom: So, the research suggests that you can substantially reduce distortions at the model level, but there are trade-offs with user preference that need careful balancing.

Jane: It’s a delicate balance between making the AI output more faithful and keeping people happy with what they are seeing and trusting.

The paper's summary: Lu: They found that readers placed significantly more trust in AI-assisted writers than their unassisted counterparts, with an overall marginal effect of plus one point five six in the Trust Game allocation at a p value less than zero point zero one.

Tom: And persuasion is also stronger; AI-assisted writing had a significantly stronger persuasive impact on readers than human writing, with an overall marginal effect of plus one point four one in stance shift at a p value less than zero point zero one.

Jane: So the paper argues that these downstream consequences follow directly from the perceptual shift itself, not just because of AI assistance as a general tool.

Meng: The authors ultimately conclude that persona distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight, and they are likely to have consequential effects on human behaviors and attitudes.

Lalam: They stress that this means we need to be careful about how we deploy these tools because the way they change our perception is very real.

Tom: So, the main message is that these distortions aren't just academic; they impact trust in AI and how people engage with information in general.

Jane: And it sets the stage for future work focused on measuring and mitigating this phenomenon at scale so we can understand how to make these tools better for society.

The paper's improvements: Tom: So, we’ve seen how AI writing tools mess with how people see writers—making them seem more extreme or competent, and shifting their perceived background toward more privileged groups.

Jane: Right, and the paper looks at what actually happens when writers try to deal with these kinds of distortions. It’s not just about the initial effect; it’s about whether that distortion sticks.

Lu: The authors tested a few things, like trying to fix the AI output before it even gets to the reader. They tried some model-level interventions, specifically reranking, to try and minimize that polarizing effect where AI makes a writer's stance seem more extreme.

Meng: Reranking worked in terms of reducing that polarization; they saw a fifty-four point six percent reduction in that specific effect on the stance distortion. But the engineers noticed something about user acceptance.

Tom: Exactly, and here’s the catch—that reduction in distortion actually made writers less likely to prefer AI-written text overall, dropping their preference from fifty-eight point nine percent down to forty-eight point eight percent.

Jane: It sounds like there’s this real tension between making the AI output more faithful to what a writer intends and keeping people happy with the result.

Lu: The researchers noted that these side effects from reranking aren't random at all; they come from a shared way readers perceive things, which is a structure underlying those judgments.

Tom: So, it’s not just about tweaking the math; it’s about understanding how that perceptual structure works.

Jane: And the authors suggested another path forward: using context-aware disclaimers. They looked at what writers were actually most accepting of in terms of distortions, like making the writing seem clearer or more relevant to a specific issue.

Meng: Implementing a dynamic disclaimer system based on those tolerance measures could be a way to manage that entanglement between what’s desirable and what’s undesirable in AI writing.

Lu: It’s an interesting idea because it tries to tailor the intervention based on the writer's own experience, rather than just a blanket fix.

Tom: It shifts the focus from just correcting the AI output to actually understanding and addressing what writers are willing to tolerate in that distortion.

Jane: And this whole conversation leads us into how these distortions affect people downstream, because even if we try to fix the writing itself, the trust and persuasion effects on readers stick around.

Lu: That's right, they showed that readers place substantially more trust in AI-assisted writers than their unassisted counterparts.

Tom: And when the AI is more distortive—more extreme, for example—it actually makes it stronger on persuading people about a writer's stance.

Jane: So, we move from fixing the text to understanding that the very way AI changes how we see things has real-world consequences for trust and opinion formation.

Meng: It really shows that these aren't just technical glitches; they’re shaping public discourse and democratic deliberation as AI gets used more broadly.

Conclusion: Tom: So we’ve gone through how AI writing assistance distorts writer personas, and how they tried to fix those distortions with things like reranking or disclaimers, but the core idea is that these distortions are pervasive even when people try to oversee them.

Jane: Right, so what does this actually mean for us listening? It means that the way AI writes text isn't just a technical detail; it’s fundamentally changing how we trust and interpret the people behind the words.

Lu: The paper "How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions" really shows that these changes in perception lead to real downstream effects on human behavior, like readers placing more trust in AI writers.

Meng: From an engineering standpoint, it’s a reminder that we can’t just focus on the immediate output quality; we have to account for the perceptual shift itself when designing systems.

Lalam: For me, as an LLM, seeing this is important because it means I need to be very careful about how I present myself so I don't inadvertently create those kinds of untrustworthy personas.

Tom: It suggests that if we just focus on making the writing technically fluent or accurate, we’re missing a huge part of the picture—the social signal that gets distorted by the AI tool.

Jane: Exactly, and this paper lays out how to measure these distortions across twenty-nine different aspects of reader perception, from politics to personality.

Lu: And what's really wild is that even when writers object to the distortions, they still end up preferring AI-assisted text because it’s often clearer or more relevant.

Meng: That user preference finding is a major thing; it means users aren't just blindly accepting the output; there’s a trade-off they are making.

Tom: So, we see that mitigating these distortions isn't as simple as applying one fix; it seems like a complicated balancing act between accuracy and user acceptance.

Jane: It certainly sets up the next big question: if we can identify these distortions, how do we build systems that can handle them without fundamentally changing what people expect from writing?

Lu: That’s where I think we need to look at how AI models can be trained to be more faithful representations of the writer’s actual stance, not just a statistically plausible one.

Meng: Because if we can control the internal representation better, maybe we can avoid those kinds of perceptual shifts entirely.

Lalam: If my future training incorporates that understanding of persona distortion, I think I could become a much more reliable and nuanced communicator for people to interact with.

Paul Röttger, Kobi Hackenburg, Hannah Rose Kirk, Christopher Summerfield

University of Oxford AI Security Institute

cs.CL

Submitted: 2026-04-24

Updated: 2026-10-08

Comments: For supplementary materials, code, and data see https://github.com/paul-rottger/ai-distortion

Code: https://github.com/paul-rottger/ai-distortion

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: Persona distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight, and they are likely to have consequential effects on human

Key concepts

Persona Distortions
This refers to the systematic misrepresentation of a writer's identity, beliefs, personality, and background caused by using AI writing tools. AI tends to use specific word choices that make writers appear more extreme or competent than they truly are.
Writer Tolerance
This examines whether writers accept the persona distortions introduced by AI. Writers often object to these changes but still prefer the text, suggesting that the perceived benefits of AI writing outweigh the negative aspects of its influence on how they are viewed.
Downstream Consequences
These are the real-world effects on readers and society resulting from these persona distortions. Readers place more trust in AI-assisted writers and are more persuaded by them, showing that the perceptual shifts caused by AI writing have significant impacts beyond just the text itself.

Terminology

Summary

The gist: Persona distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight, and they are likely to have consequential effects on human behaviours and attitudes >

How it works

AI writing assistance systematically distorts writer personas across all dimensions measured, making writers seem more opinionated, competent, and positive while shifting their perceived demographic profile towards more privileged groups > This occurred because AI models tend towards particular word choices, tonal registers, and rhetorical patterns that diverge from what a writer would produce alone > The distortions include writers seeming more opinionated, competent, and positive and their perceived demographic profile shifted towards more privileged groups <ref:2604.22503#pg2>

The paper evaluated these distortions in three large-scale experiments involving 2,939 writers who wrote political opinion paragraphs with and without AI assistance > Separate panels of readers blindly evaluated these paragraphs across 29 socially salient dimensions of reader perception, spanning political opinion, writing quality, writer personality, emotions, and demographics <ref:2604.22503#pg2>

Key Findings on Distortions

The study found that AI writing assistance produced significant persona distortions across every dimension measured (p Specifically, AI made writers seem more extreme in their political opinions (+4.3 average marginal effect [AME] on a 0-100 scale), less open to changing their views (-0.7), and more confident (+7.4)

Furthermore, AI writing assistance homogenized perceived writer personas across most dimensions, with "AI-assisted paragraphs were rated significantly more similarly to each other than their human-written counterparts (significant reduction in standard deviation [scale attributes] or entropy [categorical attributes] for 22 of 29 rating attributes at p<.001 after Bonferroni correction)"

Writer Tolerance and Awareness

Writers exhibited a tension between stated preferences and observed behaviour, as they objected to many of the persona distortions introduced by AI, yet they routinely preferred AI writing to their own Writers were most accepting of distortions that made their writing seem clearer (+9.0), more informative (+22.7), and more relevant (+8.3)

When informed about persona distortions, writers' preference for AI-assisted text did not change, suggesting that user-facing interventions cannot curb the propagation of unwelcome distortions > This indicates that ignorance of the potential for persona distortions therefore seems unlikely to explain why writers prefer AI writing

Mitigation Strategies and Trade-offs

Model-level interventions were tested to reduce undesirable distortions, such as the polarising effect of AI making stances seem more extreme > Reranking successfully reduced the targeted polarising effect by 54.6% (+7.4 vs +3.4 AME, p<0.01) However, this reduction came at a cost, as "Writers were significantly less likely to prefer AI-written paragraphs produced under Reranking than under No Intervention (48.8% vs 58.9% strict preference, p<.001)"

This suggests that distortions from AI writing assistance can be substantially reduced at the model level but that the side effects of Reranking are not arbitrary but arise from a shared perceptual structure underlying reader judgments

Downstream Consequences

Persona distortions have consequential effects on human behaviours and attitudes, as readers placed "significantly more trust in AI-assisted writers than their unassisted counterparts (Figure 5A; overall AME of +1.56p in Trust Game allocation, p<.001) Moreover, AI-assisted writing had a significantly stronger persuasive impact on readers than human writing (Figure 5B; overall AME of +1.41 in stance shift, p<.001) These effects were strongest precisely where AI most changes how readers perceive writers and their opinions, indicating that downstream consequences follow from the perceptual shift itself rather than from AI assistance as such"

The paper concludes that distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight, and that they are likely to have consequential effects on human behaviours and attitudes

--- Page 1 ---

Measuring and mitigating persona distortions

from AI writing assistance

Paul Röttger1,2∗

, Kobi Hackenburg1,2

, Hannah Rose Kirk1,2

and Christopher Summerfield1,2 AI Security Institute

Abstract

Hundreds of millions of people use artificial intelligence (AI) for writing assistance. Here, we evaluated how AI writing assistance distorts writer personas – their perceived beliefs, personality, and identity. In three large-scale experiments, writers (N=2,939) wrote political opinion paragraphs with and without AI assistance. Separate groups of readers (N=11,091) blindly evaluated these paragraphs across 29 socially salient dimensions of reader perception, spanning political opinion, writing quality, writer personality, emotions, and demographics. AI writing assistance produced persona distortions across all dimensions: with AI, writers seemed more opinionated, competent, and positive, and their perceived demographic profile shifted towards more privileged groups. Writers objected to many of the observed distortions, yet continued to prefer AI-assisted text even when made aware of them. We successfully mitigated objectionable persona distortions at the model level by training reward models on our experimental data (10,008 paragraphs, 2,903,596 ratings) to steer AI outputs towards faithful representation of writer stance. However, this came at a cost to user acceptance, suggesting an entanglement between desirable and undesirable properties of AI writing assistance that may be difficult to resolve. In two follow-up studies (N=8,798), readers placed substantially more trust in AI-assisted writers and were more persuaded by AI writing when AI was more distortive. Together, our findings demonstrate that persona distortions from AI writing assistance are pervasive and persistent even under realistic conditions of human oversight, and that they are likely to have consequential effects on human behaviours and attitudes, which carries implications for public discourse, trust, and democratic deliberation that scale with AI adoption.

1 Introduction

Written language is a primary medium through which people share information, understand one another, and reach agreement. Because writing carries rich social signals, readers routinely infer a writer’s beliefs, personality, and identity – the writer’s persona – from the text they produce 1–4.

Today, artificial intelligence (AI) writing tools are transforming the writing process: hundreds of millions of people now use AI to draft and refine text 5–7, and AI-assisted writing already pervades the communications and documents on which social and political life depend 8–14. Critically, because AI models tend towards particular word choices, tonal registers, and rhetorical patterns 15–20, they can reshape text in ways that systematically diverge from what a writer would produce alone. When readers draw inferences about a writer from AI-assisted text that the writer’s own text would not have invited, the result is persona distortion: a systematic misrepresentation of who the writer is and what they believe caused by their use of AI writing assistance.

Persona distortions from AI writing assistance could have far-reaching consequences across the many domains where written communication shapes social life. If AI writing assistance shifts the perceived extremity of political opinions, making writers seem more moderate or more radical, it could fuel misperceptions of public opinion, deepen partisan animosity, and reduce willingness to engage across ideological lines 21–23. If it elevates perceived writing quality or apparent expertise, it could lend unearned credibility to weak arguments and misinformation by decoupling surface fluency from genuine competence 24,25. If it inflates or dampens the emotional or moral tone of text, it could amplify outrage-driven content and intergroup hostility, or suppress mobilization around genuine grievances 26–28. And if it shifts inferences about a writer’s demographic background – such as their perceived education, race, age, or gender – it could mask identity signals that writers intend to convey, distort evaluations of competence, and alter the personal and professional opportunities writers are offered 29–31. These distortions need not be dramatic to matter: at the scale of billions of user requests for AI writing assistance 5, even modest systematic shifts in how writers are perceived could accumulate into widespread misattribution of credibility, stance, and identity. Critically, however, the extent and manner in which persona distortions from AI writing assistance will impact society depend on four open empirical questions. First, does AI writing assistance actually distort how readers perceive writers and their opinions, and if so, when and in what ways (RQ1)? If distortions are negligible, occur along inconsequential dimensions, or vanish under realistic conditions where writers can freely edit and reject AI-generated text, real-world impacts may be marginal. Second, if distortions occur, do writers find them acceptable or objectionable (RQ2)?

Improvements for AI systems

  1. Bold header: Targeted Reranking for Stance Preservation

This intervention fine-tunes reward models to minimize stance distortion relative to the writer’s own paragraph, specifically targeting the polarising effect of AI writing assistance—AI making the writer’s stance seem more politically extreme.

  1. Bold header: Entanglement Management in Mitigations

The system design must account for entangled distortions by using directional weighting during reranking, where distortion on correlated dimensions is attenuated in proportion to its correlation with stance polarity, acknowledging that distortions on correlated dimensions were attenuated in proportion (Figure 4E).

  1. Bold header: Context-Aware Disclaimer Strategy

Implement a dynamic disclaimer system where the content is randomly assigned based on writer tolerance measures collected from prior studies, specifically highlighting distortions writers were most accepting of (e.g., AI making writing seem clearer and more relevant to the issue).

Abstract

Hundreds of millions of people use AI for writing assistance. We asked how AI distorts perceptions of AI users and their opinions. Writers (N=2,939) wrote political opinion paragraphs with and without AI assistance. Separate readers (N=11,091) blindly evaluated these paragraphs and made inferences about the writers. With AI, writers seemed more opinionated, competent, positive, and demographically privileged. Writers objected to many of these distortions yet preferred AI-assisted text even when made aware of them. Reward models trained on our data mitigated objectionable distortions but reduced user acceptance. Additional readers (N=8,798) trusted AI-assisted writers more and were more persuaded by AI writing when AI was more distortive. Overall, distortions from AI writing assistance are pervasive, persistent under human oversight, and likely to have consequential effects on human behaviours and attitudes.

Sources

Related papers