Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
cs.CL
Submitted: 2026-10-07
Updated: 2026-10-07
Comments: Accepted to the UserSim @ NeurIPS 2026 workshop (non-archival)
License: http://creativecommons.org/licenses/by/4.0/
The gist: As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation.
Terminology
Abstract
As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation. However, the fidelity of simulated users in comparison to real human users is generally low, and typically assessed by costly, subjective LLM judges. In this pilot study, we ask whether fidelity can instead be measured deterministically by treating a user persona sociolinguistically: as a social type that emerges from observable linguistic style, rather than one predicted by labels or descriptions a model must extrapolate into behaviour. We author personas as concrete stylistic rates, which lets us transfer two established, model-free instruments -- authorship-verification stylometry and lexicon-based content analysis -- as fidelity diagnostics. We A/B-test the sociolinguistic schema against a flat descriptive baseline across five task-oriented customer-service agents. Results show that the sociolinguistic schema improves both stylistic adherence and stylometric distinguishability for most of the tested models, with a caveat that persona style fidelity does not necessarily equal persona "naturalness". We argue that a sociolinguistic approach to persona design is a promising path towards more diverse and representative user personas, and that these metrics are most valuable in an error-attribution analysis, localizing where fidelity breaks down. This is a first step towards interventions that move user simulations closer to faithful renderings of diverse and variable linguistic outputs.
Sources
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?
- Eval4Sim: An Evaluation Framework for Persona Simulation
- Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes
- RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems
- Can Multi-Agent LLMs Identify Their Peers? Stylometric Fingerprinting in Role-Constrained Political Analysis
- TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation
- Personality Expression Across Contexts: Linguistic and Behavioral Variation in LLM Agents
- Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles
- MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness
- Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
- LoCar: Localization-Aware Evaluation of In-Vehicle Assistants through Fine-Grained Sociolinguistic Control
- MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI
- Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing
- MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
- Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
- Role Steering of Language Models for Social Simulations
- CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering