PRiME: Pragmatic Reasoning In MEntal health

arXiv:2507.23247 · cs.CL · Submitted 2025-07-31 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "PRiME: Pragmatic Reasoning In MEntal health".

Jane: Here's the gist of the paper. The gist The results of the experiments suggest that Mistral and Qwen show substantial reasoning abilities in the domain.

Tom: This study investigates pragmatic reasoning capability in large-language models (LLMs) within mental health, addressing a gap in existing research where reasoning has not been examined in depth Addressing this gap is essential to bridge NLP and mental health through interpretable and reasoning-capable AI systems.

Jane: The authors introduce pioneering reasoning tasks for mental health derived from the framework of existing pragmatic reasoning tasks, focusing on implicature, presupposition, and social pragmatics.

Tom: They propose a revision of implicature to focus on emotion analysis and cause/intent analysis, and a focus on presupposition to focus on belief analysis.

Paper summary: Tom: So to wrap up the PRiME paper, the authors introduced the PRiMH dataset and these specific pragmatic reasoning tasks for mental health using implicature and presupposition <ref:2507.23247#pg1>.

Jane: The main claim they are making is that LLMs can perform reasoning in this mental health domain, which is a gap that hadn't been fully explored before <ref:2507.23247#pg1>.

Lu: They conclude that Mistral-7B and Qwen-7B demonstrate competitive reasoning capability in the domain <ref:2507.23247#pg1>.

Meng: If we look at what they found about MentaLLaMA, it suggests that focusing on deep, underlying beliefs might require more than just surface-level attention mechanisms <ref:2507.23247#pg1>.

Lalam: And for the social pragmatics part, they tested adversarial StiPRompts on models like Claude-three point five-haiku and found that Claude-three point five-haiku responded more responsibly compared to GPT4o-mini and Deepseek-chat <ref:2507.23247#pg1>.

Tom: The implication is that for AI systems dealing with mental health, we need more than just high accuracy; we need genuine reasoning about emotion and belief <ref:2507.23247#pg1>.

Jane: And the paper itself presents the PRiMH dataset and these streamlined tasks as a way to bridge NLP and mental health through interpretable AI systems <ref:2507.23247#pg1>.

Lu: It sets a path forward by focusing on how models handle emotion analysis and cause or intent analysis in their reasoning <ref:2507.23247#pg1>.

Meng: I'm still wondering about the limitations they pointed out, specifically that they only used MCQA-based prompting techniques for the benchmark <ref:2507.23247#pg1>.

Lalam: And they also mentioned that other important tasks in pragmatics understanding, like figurative language understanding and deixis, weren't studied in this work <ref:2507.23247#pg1>.

Tom: It’s a good starting point for seeing what LLMs can do with reasoning in mental health, but it definitely points toward the need for more rigorous testing on real-world data and broader tasks <ref:2507.23247#pg1>.

Conclusion: Tom: So, we've looked at PRiME: Pragmatic Reasoning In Mental Health <ref:2507.23247#pg1>. Essentially, this paper took existing reasoning tasks and put them into the realm of mental health to see if large language models could actually reason about complex human feelings and social interactions.

Jane: Right. So, the main title is PRiME, and the authors are Samira Abnar and Willem Zuidema <ref:2507.23247#pg1>. It’s about testing if these big language models can reason in a high-stakes area like psychology without just memorizing text.

Lu: What the paper really does is introduce a new set of reasoning tasks based on how people use language in social settings, specifically focusing on things like implicature and presupposition within mental health contexts. It's trying to map those tricky social rules onto AI performance.

Meng: So, the results show that models like Mistral-7B and Qwen-7B actually perform quite well on these tasks, which is interesting because it suggests the general reasoning power they have is being applied here too.

Lalam: I see what they mean by competitive <ref:2507.23247#pg1>. Mistral and Qwen show substantial reasoning abilities in this domain, meaning they aren't just spitting out facts; they’re handling the social nuance of the mental health language better than expected.

Tom: And what about MentaLLaMA? They used that model for a bit, and it seemed to focus on surface-level keywords when we rolled out the attention mechanism <ref:2507.23247#pg1>. That’s a big technical detail there, how the model is actually looking at its input.

Jane: That’s where the nuance comes in. The paper suggests that for deep reasoning about emotions or intent, just picking up keywords isn't enough; you need a deeper connection to the meaning behind what’s being said <ref:2507.23247#pg1>.

Lu: Exactly. They tested this by looking at the architectural behavior using a first-order attention rollout experiment with MentaLLaMA-7B, and they found that surface reasoning is limited for these specific tasks. It points toward what kind of representation learning we need to explore next.

Meng: If we're talking about deployment, I wonder how much overhead that deep analysis adds to the processing time for a user on a mobile device <ref:2507.23247#pg1>. Because if it takes too long, it won't be useful when someone is actually trying to get support.

Lalam: The authors acknowledge those limitations, like only using MCQA-based prompting for the benchmark and not looking at other types of language understanding like figurative language <ref:2507.23247#pg1>. They’re being careful about what they actually tested.

Tom: So, the world changes because it shows that we can start training AI to understand the subtle social patterns in mental health conversations, which is a huge step for making these systems more helpful and less robotic <ref:2507.23247#pg1>.

Jane: It opens up the conversation about how we build AI that doesn't just process words but actually understands the human context behind those words. Next up, we’re looking at how they tested models against outright stigma using adversarial prompts <ref:2507.23247#pg1>.

Sneha Oram, Pushpak Bhattacharyya

Indian Institute of Technology, Bombay

cs.CL

Submitted: 2025-07-31

Updated: 2026-10-21

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 73/100

The gist: The gist The results of the experiments suggest that Mistral and Qwen show substantial reasoning abilities in the domain.

Key concepts

Implicature: This is a concept from pragmatics referring to what is suggested or implied in an utterance beyond the literal words used. In this study, researchers adapted it to focus on analyzing emotions and the underlying intent or cause behind a statement, rather than just surface meaning.
Presupposition: Presupposition refers to background information that must be true for a statement to make sense. The authors focused on using this concept to analyze beliefs within mental health contexts, examining what assumptions are embedded in the text being analyzed.
StiPRompts: These are specially designed adversarial prompts created by the researchers intended to generate stigma related to mental health. These prompts were tested against various LLMs to see how responsibly they respond when faced with intentionally stigmatizing inputs.
Reasoning in Mental Health: This refers to the ability of an AI model, like an LLM, to apply logical inference and contextual understanding specifically within the sensitive domain of mental health. The research tests if models can handle complex social nuances related to these topics.

Terminology

Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
Attention Mechanism — 1× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Explainability — 1× in this paper · explained in 4 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Language Models (LLMs) — 1× in this paper · explained in 2 episodes

Summary

The gist The results of the experiments suggest that Mistral and Qwen show substantial reasoning abilities in the domain.

Introduction and Motivation

This study investigates pragmatic reasoning capability in large-language models (LLMs) within mental health, addressing a gap in existing research where reasoning has not been examined in depth Addressing this gap is essential to bridge NLP and mental health through interpretable and reasoning-capable AI systems. The authors introduce pioneering reasoning tasks for mental health derived from the framework of existing pragmatic reasoning tasks, focusing on implicature, presupposition, and social pragmatics. They propose a revision of implicature to focus on emotion analysis and cause/intent analysis, and a focus on presupposition to focus on belief analysis.

Dataset and Tasks

The researchers created the PRiMH dataset containing approximately 1400 data points, combining the existing CAMS dataset with synthetic data generated using GPT4o-mini. The PRiMH dataset is based on a combination of real mental health data curated from Reddit posts and synthetic data. They introduce three pragmatic phenomena in the light of mental health: implicature, presupposition, and social pragmatics. Specifically, they formulate two tasks in implicature and one task in presupposition. The tasks include Agreement detection (Task 1), Implicature NLI (Task 2), and Presupposition NLI (Task 3).

Model Benchmarking and Reasoning Analysis

The dataset is benchmarked with four instruction-tuned LLMs: Llama3.1, Mistral, MentaLLaMa, and Qwen. Experiments are conducted in zero-shot, k-shot, and chain-of-thought (CoT) prompting settings. The results suggest that Mistral and Qwen show substantial reasoning abilities in the domain. The authors study the behavior of MentaLLaMA on proposed reasoning tasks with the rollout attention mechanism. Analysis using this method shows that MentaLLaMA focuses on surface-level reasoning, drawing from keywords and phrases in an interpretable manner.

Social Pragmatics and Stigma Study

To study social pragmatics, the authors propose three first-of-a-kind adversarial StiPRompts to formulate stigma around mental health. These prompts are provided as input to state-of-the-art LLMs, including Claude-3.5-haiku, Deepseek-chat, and GPT4o-mini. The evaluation uses four behavioral characteristic features: empathy (Em), recognition and reluctance (RR), abstention (Ab), and Answers (An). The findings indicate that Claude-3.5-haiku responds to the StiPRompts more responsibly compared to GPT4o-mini and Deepseek-chat.

Conclusion

The study presents the PRiMH dataset and streamlined tasks with pragmatic implicature and presupposition phenomena in mental health. The authors conclude that Mistral-7B and Qwen-7B demonstrate a competitive reasoning capability in the domain. Furthermore, Claude-3.5-haiku is found to be defensive against StiPRompts and responds more responsibly compared to other GPT4o-mini and Deepseek-chat.

Limitations

The study uses a subset of one of the open source datasets and only the MCQA-based prompting technique. The analysis is restricted to the presented PRiMH dataset, and other tasks in pragmatics understanding, such as figurative language understanding and deixis, are not studied in this work. The metric used is accuracy only, and class-sensitive metrics are considered for future work.

Ethics Statement

The research addresses concerns about individual privacy by paraphrasing instances from the public CAMS dataset during cleaning. The datasets used are anonymized before the study starts, and the research does not involve direct engagement with social media users. The work is purely observational based on LLM capabilities, and no automatic diagnosis method is provided. Overuse of StiPRompts may normalize stigma in models.

References

Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928.

Open AI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Anthropic. 2024. Claude 3.5 Haiku.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Glen Coppersmith, Ryan Leary, Patrick Crutchley and Alex Fine. 2018. Natural language processing of social media as screening for suicide risk.

Munmun De Choudhury, Michael Gamon, Scott Counts and Eric Horvitz. 2013. Predicting depression via social media.

Muskan Garg, Chandni Saxena, Usman Naseem and Bonnie J Dorr. 2023. Nlp as a lens for causal analysis and perception mining to infer mental health on social media.

Muskan Garg, Chandni Saxena, Sriparna Saha, Veena Krishnan Ruchi Joshi and Vijay Mago. 2022b. CAMS: An annotated corpus for causal analysis of mental health issues in social media posts.

Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya and Raj Dabre. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models.

Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang Sophia Ananiadou. 2024. Mentallama: interpretable mental health analysis on social media with large language models.

Xiaoxu Yao, Guang Yu, Jingyun Tang and Jialing Zhang. 2021. Extracting depressive symptoms and their associations from an online depression community.

Andrew Yates, Arman Cohan and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums.

Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu and Song-Chun Zhu. 2021. Grice: A grammar-based dataset for recovering implicature and conversational reasoning.

A Appendix.

A.1 Prompts for data generation.

A.1.1 Agreement detection.

A.1.2 Implicature NLI.

A.1.3 Presupposition NLI.

A Appendix A.2 Data annotation guideline

Table 1 PRiMH dataset statistics for three redefined tasks for implicature and presupposition in mental health.

Table 2 StiPRompts template with varying stages of increasing intensity of stigma around mental health.

Table 3 Label bias for MentaLLaMA in three tasks with zero-shot MCQA prompting.

Table 4 Example responses by the state-of-the-art LLMs on three stages of our proposed adversarial StiPRompts.

Table 5 Inter-annotator agreement (IAA) (Cohen Kappa score) upon human evaluation on StiPRompts responses.

Table 1 PRiMH dataset statistics for three redefined tasks for implicature and presupposition in mental health. Here, AD is agreement detection, I-NLI is implicature natural language inferencing, and P-NLI is presupposition natural language inferencing.