Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

cs.CY, cs.AI, cs.CL, stat.AP

Submitted: 2026-07-06

Updated: 2026-10-10

Comments: v2: corrects reverse-keyed item scoring for LLM answers, the distribution component of the statistical baselines, the acquiescence index, and the sign of paired differences in the counterfactual swaps (the role swap now has the largest effect), and drops three part-whole paths from the mediation negative control; all numbers, tables and figures recomputed

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• Towards Measuring the Representation of Subjective Global Opinions in Language Models
• Large language models that replace human participants can harmfully misportray and flatten identity groups

Related papers

• Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus
• Generative AI Purpose-built for Social and Mental Health: A Real-World Pilot
• Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports
• PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks
• What is an intelligent system?
• AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study