Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts

arXiv:2512.08814 · cs.CL · Submitted 2025-12-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Ask, Answer, and Detect".

Jane: Here's the gist of the paper. The gist Understanding human personality is crucial for web applications such as personalized recommendation and mental health assessment, and ROME proposes a novel framework that explicitly injects psychological knowledge into personality detection by leveraging LLMs' role-play capability to simulate user responses to validated psychometric questionnaires.

Tom: How it works ROME introduces a modeling paradigm illustrated in Figure 1(c) where the key idea is to treat the standardized personality questionnaire as a structured retrieval source and design a label-aware LLM agent to role-play each user, generating question-level answers from their free-form posts.

Jane: These generated answers constitute questionnaire-grounded evidence that explicitly links linguistic cues to specific personality labels.

Tom: The framework comprises three key stages: Ask, Answer, and Detect.

Paper summary: Tom: So, to recap, the paper "Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts" proposes a framework that uses LLM role-playing to simulate user responses to validated psychometric questionnaires. The core claim is that injecting this psychological knowledge helps overcome the limitations of existing methods that rely only on limited supervision from coarse personality labels <ref:2512.08814#pg2>.

Jane: Right. They are tackling the problem where we don't have enough labels to train a model reliably, and they suggest using role-play prompting to generate question-level answers from user posts, creating intermediate supervision that links language directly to specific personality constructs <ref:2512.08814#pg3>.

Lu: The framework is structured in three stages: Ask, Answer, and Detect. The key idea is treating the questionnaire as a structured retrieval source for the role-play agent <ref:2512.08814#pg3>.

Meng: And then they use a question-conditioned Mixture-of-Experts module to route the post and question representations to predict those answers under explicit supervision <ref:2512.08814#pg3>.

Tom: The final step is combining that questionnaire evidence with the user representation from their posts in the Detect stage for the final personality prediction <ref:2512.08814#pg3>.

Jane: It’s about making the model's predictions based on answers to specific questions, which makes them much more interpretable than just a raw label guess <ref:2512.08814#pg3>.

Lu: And they found that this method outperforms state-of-the-art baselines on two real-world datasets, showing relative improvements of thirteen point six zero percent on Kaggle and nineteen point seven two percent on Pandora <ref:2512.08814#pg1>.

Meng: So, the numbers are there, and they show a tangible performance boost over what was previously available in this area <ref:2512.08814#pg1>.

Tom: It really validates the idea that explicitly injecting psychological knowledge through structured inputs can make LLM-based personality detection much stronger than just relying on post encoding alone <ref:2512.08814#pg3>.

Jane: So, the takeaway is that this framework offers a way to get more reliable and interpretable personality predictions by grounding them in standardized psychological assessments <ref:2512.08814#pg3>.

Lu: This moves the goal from just encoding text to explicitly modeling the mapping from language to psychology using structured knowledge, which is a key direction for representation learning <ref:2512.08814#pg3>.

Conclusion: Tom: So we’ve covered how ROME works and what the results show in "Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts." The authors are Yifan Lyu and Liang Zhang <ref:2512.08814#pg1>.

Jane: They’ve shown that by using role-playing to simulate questionnaire responses, they can create a framework that gives intermediate supervision, which helps them build better text-to-personality mappings <ref:2512.08814#pg3>.

Lu: The paper suggests this is a direction for representation learning where we stop treating the mapping as purely learned from noisy, sparse labels and start grounding it in known psychological structures <ref:2512.08814#pg3>.

Meng: For practical terms, it means that when we are building systems that need to understand user behavior for things like recommendations or mental health checks, we should look at using structured inputs like questionnaires to give the AI a more reliable signal <ref:2512.08814#pg3>.

Tom: The overall implication is that this Ask–Answer–Detect framework provides a way to extract robust, psychology-aligned evidence even when the input data might be incomplete or from sparse sources <ref:2512.08814#pg3>.

Jane: It’s about creating systems where we can not only predict a label but also show the evidence—the simulated answers—that led to that prediction, which is a big step for trust in AI systems <ref:2512.08814#pg3>.

Yifan Lyu, Liang Zhang

The Hong Kong University of Science and Technology (Guangzhou) · University of International Business and Economics

cs.CL

Submitted: 2025-12-09

Updated: 2026-10-20

Comments: Accepted at NeurIPS 2026 (Main Conference, Poster)

Code: https://github.com/JasmineLyf/ROME

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist Understanding human personality is crucial for web applications such as personalized recommendation and mental health assessment, and ROME proposes a novel framework that explicitly injects

Key concepts

Role-Play-Based Questionnaire Response Generation: An LLM agent is instructed to act as a user and answer each item of a standardized personality questionnaire based on their posts and a given personality label. This generates 'questionnaire-grounded evidence' that explicitly connects linguistic patterns to specific psychological traits, helping the model learn the mapping function.
Question-Conditioned Mixture-of-Experts (MoE): This architecture uses multiple specialized experts within an LLM structure. These experts focus on capturing different psychological constructs and their associated linguistic cues. A shared expert handles general knowledge, and the final answer is a weighted combination of these specialized outputs, allowing for nuanced prediction.
Psychological Evidence-Guided Prediction: The final personality prediction combines the user's post representation with the predicted answers from the questionnaire. The contribution of each predicted answer is adaptively weighted based on its reliability and importance scores derived from the questions, ensuring that semantically aligned and psychologically meaningful signals drive the final result.

Terminology

Language Models (LLMs) — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist Understanding human personality is crucial for web applications such as personalized recommendation and mental health assessment, and ROME proposes a novel framework that explicitly injects psychological knowledge into personality detection by leveraging LLMs' role-play capability to simulate user responses to validated psychometric questionnaires.

How it works

ROME introduces a modeling paradigm illustrated in Figure 1(c) where the key idea is to treat the standardized personality questionnaire as a structured retrieval source and design a label-aware LLM agent to role-play each user, generating question-level answers from their free-form posts. These generated answers constitute questionnaire-grounded evidence that explicitly links linguistic cues to specific personality labels. The framework comprises three key stages: Ask, Answer, and Detect.

Ask: Role-Play-Based Questionnaire Response Generation

In the Ask stage, a role-play LLM agent is instructed to “take on the perspective of the user" and produce answers to each questionnaire item 1 based on the user’s posts and the corresponding personality label. The posts provide evidence of the user’s linguistic patterns and behavioral tendencies, while the label supplies a stable trait orientation. To ease the learning of the mapping function f (·), ROME introduces standardized psychometric questionnaires to decompose each personality label into multiple semantically meaningful questions. The LLM agent outputs a score on an ordered scale that approximates how user u would answer Qi. Since LLM generation is inherently stochastic, the model draws T samples and averages them to reduce sampling variance and obtain a more robust estimate.

Answer: Question-Conditioned Mixture-of-Experts Framework

The goal of the Answer stage is to enable the model to reproduce these answers directly from the user posts and the questionnaire item texts. To address limitations of a single shared encoder, ROME proposes a question-conditioned Mixture-of-Experts (MoE) architecture. This MoE structure allows different experts to specialize in capturing distinct psychological constructs and their associated linguistic cues, while still enabling shared experts to model knowledge that is broadly useful across questions. The final answer is then predicted as a weighted sum of expert outputs.

Detect: Psychological Evidence-Guided Personality Prediction

In the Detect stage, the questionnaire-grounded evidence is aggregated with the user representation learned from posts to perform the final prediction. The contribution of each predicted answer is adaptively weighted based on its reliability and importance. The adaptive weight w i is computed as qimp i · qrel i, where qimp i is the importance score associated with question Qi, and qrel i is the reliability score associated with question Qi. The final answer-based representation s u is obtained by element-wise multiplication of the weights and predicted answers.

Overall Results and Contributions

Extensive experiments on two real-world datasets demonstrate that ROME consistently outperforms state-of-the-art baselines, achieving relative improvements of up to 15.41%. ROME is the first work to leverage LLM’s role-play capabilities to explicitly inject psychological knowledge into personality detection and formulate an interpretable, questionnaire-grounded modeling paradigm.

Ablation Study Insights

The ablation study on Kaggle dataset indicates that removing reliability and importance-based weights reduces the average Macro-F1 from 89.78% to 88.47%. This degradation reflects the heterogeneity across questionnaire items. Furthermore, using only post representations without evidence leads to a drop in Macro-F1 to 65.77%, indicating that post-derived features capture broad and expressive linguistic cues while questionnaire-grounded evidence provides semantically aligned and psychologically meaningful signals.

Conclusion

ROME performs best when both sources are jointly leveraged, indicating that these two perspectives are complementary rather than interchangeable. The model remains highly competitive even in the most data-poor regime with only 40% of the training data, maintaining a strong performance. The Ask–Answer–Detect framework benefits from a much denser and more structured training than traditional “posts → labels” models. The paper demonstrates that the Ask–Answer–Detect framework can extract robust, psychology-aligned evidence even from incomplete questionnaires.

--- Page 1 ---

Ask, Answer, and Detect: Role-Playing LLMs for Personality Detection with Question-Conditioned Mixture-of-Experts arXiv’25, 2025

Improvements for AI systems

Bold Header: Role-Play Enhanced Evidence Generation

The improved system can leverage LLMs’ role-play capability to simulate user responses to validated psychometric questionnaires offline during training, transforming free-form posts into questionnaire-grounded evidence that explicitly links linguistic cues to specific personality labels.

Bold Header: Question-Conditioned Mixture-of-Experts Routing

The system will utilize a question-conditioned Mixture-of Experts (MoE) module that jointly routes over post and question representations, learning to answer questionnaire items under explicit supervision, allowing experts to specialize in distinct psychological constructs and their associated linguistic cues.

Bold Header: Adaptive Evidence Weighting for Prediction

The final prediction will be formed by a mechanism where the final weight is computed as wi = qimpi · qreli, ensuring that more important and more reliable questions should contribute more to the final prediction, effectively balancing the contributions of post-derived representations and questionnaire evidence.

Bold Header: Multi-Task Learning Framework

The system will employ a multi-task learning framework where question answering serves as a powerful auxiliary task for personality detection, optimizing both the question prediction loss (Lq) and the final personality classification loss (Lcls) jointly.

Bold Header: Interpretability via Evidence Chains

The resulting system will provide an interpretable, question-grounded modeling paradigm and a semantic reasoning chain that guides and simplifies the text-to-personality mapping learning, allowing for analysis of how specific questionnaire items align with semantically relevant post snippets.