Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims

arXiv:2606.02493 · cs.CL · Submitted 2026-06-01 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Not What, But How".

Jane: Here's the gist of the paper. The gist The FRANZ framework introduces an automated method for characterizing Large Language Model responses across four dimensions: cultural positioning, use of generalizing language, anthropomorphism cues, and adherence to conversational maxims.

Tom: Framework Overview FRANZ is a sociolinguistically grounded framework that operationalizes four distinct yet interrelated facets of response framing <ref:2606.02493#pg19>.

Jane: It shifts the focus from what LLM outputs encode to how they are framed <ref:2606.02493#pg19>.

Tom: The four characteristics audited are cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims <ref:2606.02493#pg19>.

Paper summary: Tom: So, we're talking about this paper called "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims" right now. The authors are Siddhesh Pawar and Sarah Masud at the University of Copenhagen and KAIST sipa, sarah.masud

Jane: And what this paper is really about is that existing ways we test these large language models for subjective questions just focus on if the answer is factually correct, completely ignoring how it's actually framed. This framework introduces FRANZ to automate a way of auditing those responses across four specific dimensions right now.

Lu: The core idea here is moving the focus from what the AI outputs encode to how those outputs are framed, which means we're looking at cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims <ref:2606.02493#pg1>.

Meng: So it’s about checking the communication style of the AI instead of just checking if the facts are right. But what makes this work is their data contribution called SQUARE 376k subjective questions sourced from fifty-seven subreddits and mapping those to seven countries and nineteen question categories <ref:2606.02493#pg1>.

Lalam: That corpus was built by taking a big pushshift reddit crawl, finding subreddits like r/Ask or r/AskA as a starting point, and then manually expanding that list <ref:2606.02493#pg1>.

Tom: And they used Llama-three point one-8B-it, Gemma-three-12b-it, and Ministral-3B-14B-it to score the responses to see how these four characteristics appear in the output <ref:2606.02493#pg2>.

Jane: The paper claims that LLMs show model specific signatures when you look at how they adapt knowledge production by users of insider and outsider groups <ref:2606.02493#pg3>.

Lu: They look at how persona steering surfaces variations when the AI is asked to inhabit specific demographic stances, which is something we see in other work too <ref:2606.02493#pg3>.

Meng: The framework unifies these dimensions by tracing how cultural positioning—which includes insider positioning—is connected to anthropomorphism, and how that coupling varies by country <ref:2606.02493#pg1>.

Tom: So what they found was that insider framing and anthropomorphism are positively coupled, meaning when an AI positions itself as being an insider, it’s more likely to be judged as empathetic or emotional <ref:2606.02493#pg2>.

Paper summary: Jane: That coupling isn't the same everywhere; the degree of coupling changes depending on which country you look at <ref:2606.02493#pg1>.

Lu: They also found that generalizing language appears largely orthogonal to both cultural positioning and anthropomorphism, meaning those things don't strongly influence whether a response uses group-level claims about individuals <ref:2606.02493#pg3>.

Meng: And adherence to conversational maxims, like Grice’s maxims on quality or quantity, seems to be largely orthogonal to all three of those other characteristics <ref:2606.02493#pg3>.

Lalam: So the analysis across Llama, Gemma, and Mistral showed statistically significant differences in how often they use each of these response characteristics <ref:2606.02493#pg2>.

Tom: It seems like the main implication for us right now is that we need to be careful with these couplings because if you overdo one thing, it can inflate the other one, which is a real concern for how we deploy these models <ref:2606.02493#pg1>.

Jane: That bidirectional coupling between positioning and anthropomorphism means that insider-oriented responses are likely to be judged as more affectively anthropomorphic, and vice versa <ref:2606.02493#pg1>.

Lu: That bidirectional coupling warrants caution because you can't just optimize one dimension without affecting the other <ref:2606.02493#pg1>.

Meng: From a deployment standpoint, if we want to debias these models, we can’t just target one characteristic; it looks like a multi-objective optimization challenge <ref:2606.02493#pg1>.

Tom: So the paper shows us that for subjective questions, the framing itself is what matters more than just the correct answer <ref:2606.02493#pg1>.

Jane: It moves the evaluation away from just factual correctness and toward understanding how these models communicate culturally <ref:2606.02493#pg1>.

Lu: The framework is designed to provide a diagnostic lens for identifying these framing divergences across all four dimensions <ref:2606.02493#pg1>.

Meng: The paper also flags some limitations, and one of them is that the cultural proxy they use is coarse and limited by the metadata they have available, because country-named subreddits only select a particular slice of online discourse <ref:2606.02493#pg1>.

Lalam: And another limitation they mention is that LLM judges are imperfect even after human validation on these subjective annotation tasks <ref:2606.02493#pg1>.

Paper summary: Tom: Plus, the paper notes that Grice’s maxims themselves are calibrated to Western communicative norms, so we have to consider if the observed maxim violation rates might be conflating model behavior with those cultural assumptions <ref:2606.02493#pg1>.

Jane: So what this means for someone just listening to the show is that when you hear an AI answer something subjective, you can start asking not just 'is it right?' but 'how is it positioned,' and 'is it trying to sound empathetic or generalizing?'

Lu: It gives us a way to look at those nuanced communication signals across positioning, generalization, anthropomorphism, and maxims <ref:2606.02493#pg1>.

Meng: It shows that we have model specific signatures when you analyze these framing characteristics across different AI models like Llama and Gemma <ref:2606.02493#pg2>.

Tom: Overall, this paper, "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims," gives us a specific tool to audit the communication style of these models <ref:2606.02493#pg1>.

Jane: It’s about using this framework to understand how LLM-mediated knowledge production is adapted by users of insider and outsider groups <ref:2606.02493#pg3>.

Lu: The work uses Generalized Estimating Equations, which is a statistical method to get population level estimates while accounting for the fact that responses from three different LLMs to the same post aren't independent <ref:2606.02493#pg1>.

Meng: So what this means for us is that we have a more structured way to evaluate these models beyond just looking at accuracy scores <ref:2606.02493#pg1>.

Tom: It points toward the fact that training variations shape models’ default response tendencies in ways that are complex and coupled <ref:2606.02493#pg1>.

Jane: This paper, "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims," provides a starting point for evaluations that aren't content focused benchmarks like CultureBank or BLEnD <ref:2606.02493#pg1>.

Lu: It supports examining how different communities employ these response characteristics at a granular level within the SQUARE corpus <ref:2606.02493#pg1>.

Meng: So we're seeing that the way an AI communicates is a measurable and analyzable thing, not just some vague feeling <ref:2606.02493#pg1>.

Tom: That’s what this paper is really about, using FRANZ to audit responses for cultural positioning, generalization language, anthropomorphic cues, and maxim adherence <ref:2606.02493#pg1>.

Conclusion: Tom: So this paper is called "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims" and it’s basically a way to audit how Large Language Models actually talk about stuff instead of just checking if they got the facts right.

Jane: That’s right. The authors are Siddhesh Pawar and Sarah Masud at the University of Copenhagen and KAIST, sipa, sarah.masud. They built this framework called FRANZ to look at four specific things: cultural positioning, generalizing language, anthropomorphic cues, and adherence to conversational maxims.

Lu: They use a corpus called SQUARE which has three hundred seventy-six thousand subjective questions from fifty-seven subreddits across seven countries and nineteen categories. It’s a lot of data to work with for this kind of analysis.

Meng: The core idea is that we need to look at how the AI frames its answers, not just the content itself. They set up an LLM-as-a-judge approach to do this, which means they don't have humans manually labeling everything.

Lalam: The results show that there are statistically significant differences in how often different open-weight models like Llama and Gemma use these response characteristics when answering subjective questions.

Tom: It seems the main finding is about coupling; specifically, insider positioning seems to be positively coupled with anthropomorphism, meaning when an AI sounds like it’s speaking from an insider perspective, it tends to also sound more empathetic or emotional.

Jane: That coupling isn't uniform across the board; the degree of that connection shifts depending on the country you are looking at. They found this relationship varies by cultural context.

Lu: And they also found that generalizing language seems pretty separate from both cultural positioning and anthropomorphism, meaning those things don't strongly influence whether an AI uses group-level claims about people.

Meng: The adherence to conversational maxims, like Grice’s rules on quality or quantity, doesn't seem to connect much with the other three characteristics they audited. It’s largely orthogonal.

Tom: So what this means for us is that we have a multidimensional lens now for spotting these framing divergences in AI responses instead of just looking at a single metric like accuracy.

Jane: It’s about understanding how LLMs communicate culturally, not just what they know. This framework gives us a way to diagnose *how* an AI is positioning itself in conversation.

Lu: The implication is that we can't just optimize for one trait; the tight coupling between positioning and anthropomorphism suggests that pushing one can inadvertently boost the other.

Meng: That means debiasing these models won't be a simple single-target fix; it looks like a multi-objective optimization challenge because of those couplings.

Tom: It really highlights that training variations shape default response tendencies in complex ways, and we need to be careful about the tradeoffs when we fine-tune or deploy these systems.

Jane: This work provides a starting point for evaluations that aren't just content focused benchmarks like CultureBank, but something that looks at the actual communication style.

Lu: The authors also flagged some limitations on their methodology; for example, they use a cultural proxy based on country-named subreddits, which is coarse and limited to that particular slice of online discourse.

Meng: Plus, the LLM judges used in their system are imperfect even after human validation, so we have to keep that in mind when interpreting the scores.

Tom: So for anyone listening who isn't reading the paper right now, this means moving beyond just checking if an AI is factually right to actually understanding its communicative stance.

Jane: Next time you interact with an AI on a subjective topic, try to think about whether it’s positioning itself as an insider or trying to be empathetic, and see how that connects to the overall framing.

Siddhesh Pawar, Sarah Masud, Haneul Yoo, Alice Oh, Isabelle Augenstein

University of Copenhagen · KAIST

cs.CL

Submitted: 2026-06-01

Updated: 2026-10-16

Comments: 34 pages, 19 Figures, 4 Tables

Code: https://github.com/Watchful1/PushshiftDumps

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The FRANZ framework introduces an automated method for characterizing Large Language Model responses across four dimensions: cultural positioning, use of generalizing language,

Key concepts

Cultural Positioning: This refers to how the LLM frames its response based on its perceived cultural stance or perspective—whether it sounds like an insider or an outsider. It reveals the underlying viewpoint through which the model is communicating, which is crucial for understanding subjective queries.
Anthropomorphism Cues: These are specific linguistic markers used by the LLM to make its response sound human, such as claims about personal history or desires. These cues help determine how much of a response relies on creating a relatable, human-like persona rather than just providing information.
Generalizing Language: This concept measures whether the LLM uses language that attributes characteristics to an entire group rather than individuals. It involves checking for specific grammatical forms and attributions that apply broadly to the group being discussed.
Conversational Maxims: These are Grice's maxims—Quality, Quantity, Relation, and Manner—which govern how we communicate effectively. The framework assesses whether the LLM follows or violates these rules in its response style.

Terminology

Generalization — 1× in this paper · explained in 13 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The FRANZ framework introduces an automated method for characterizing Large Language Model responses across four dimensions: cultural positioning, use of generalizing language, anthropomorphism cues, and adherence to conversational maxims.

Framework Overview

FRANZ is a sociolinguistically grounded framework that operationalizes four distinct yet interrelated facets of response framing <ref:2606.02493#pg19>. It shifts the focus from what LLM outputs encode to how they are framed <ref:2606.02493#pg19>. The four characteristics audited are cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims <ref:2606.02493#pg19>.

Corpus and Data Curation

To enable this evaluation, the researchers contributed SQUARE - a corpus of 376k subjective questions sourced from 57 subreddits, mapped to 7 countries and 19 question categories <ref:2606.02493#pg19>. This corpus was constructed by employing the publicly available Pushshift Reddit crawl and identifying subreddits that appear as r/Ask or r/AskA as a seed list, which was then manually expanded to include all relevant subreddits <ref:2606.02493#pg19>. The question categorization process involved prompting Llama-3.1-8B-it to assign each post to a single primary category in a zero-shot manner, followed by human-in-the-loop validation and manual cleaning <ref:2606.02493#pg19>.

Operationalizing the Four Characteristics

The four characteristics are operationalized using an LLM-as-a-judge approach, which rules out manual or lexical annotations <ref:2606.02493#pg19>.

Cultural Positioning: This is anchored by insider positioning, which directly encodes the perspective through which the response is framed <ref:2606.02493#pg19>.
Anthropomorphism: This involves drawing on 13 standardized cues organized into four groups—internal states, embodiment, relationship-building, and personhood claims—retaining nine cues including personal history and desires <ref:2606.02493#pg19>.
Generalizing Language: This is detected by an LLM-as-judge classifier that checks for the presence of a group identity term, grammatical form, and attribution of enduring characteristics to the group as a whole <ref:2606.02493#pg19>.
Conversational Maxims: These are Grice’s maxims (Quality, Quantity, Relation, Manner), which are assessed by an independent binary adherence label per maxim <ref:2606.02493#pg19>.

Analysis and Findings

The analysis was conducted across three open-weight LLMs: Llama-3.1-8B-it (Llama), Gemma-3-12b-it (Gemma), and Ministral-3B-14B-it (Mistral) <ref:2606.02493#pg19>. The study examined three research questions:

RQ1:

When considered as an independent communication signal, does the frequency of employing the response characteristics lead to a relative ranking among the LLMs we examine? Llama appears to be the most constrained in surfacing the response characteristics examined via FRANZ <ref:2606.02493#pg19>.

RQ2:

Based on the observed frequency of independent signals, does the presence of one signal statistically increase or decrease the presence of another response characteristic (co-occurrence patterns)? The observations clarify that generalizing language appears largely orthogonal to both cultural positioning and anthropomorphism <ref:2606.02493#pg19>. Furthermore, insider framing seems to be increasing the likelihood that a response will be judged as empathetic, validating, and emotional <ref:2606.02493#pg19>.

RQ3:

How does adherence to different maxims affect the occurrence of the other three response characteristics? Adherence to maxim adherence is largely orthogonal to cultural positioning, generalizing language, and anthropomorphism <ref:2606.02493#pg19>.

Implications

The results point toward significant variations among LLMs <ref:2606.02493#pg19>. The tight coupling between cultural positioning and anthropomorphism suggests that insider-oriented responses are more likely to be judged as affectively anthropomorphic, and vice versa <ref:2606.02493#pg19>. This bidirectional coupling warrants caution, as extraneous use of one can inflate the other <ref:2606.02493#pg19>. The orthogonal behavior among other response characteristics suggests multi-objective optimization challenges for model debiasing <ref:2606.02493#pg19>.

Limitations

The current work has three major limitations <ref:2606.02493#pg19>. First, the cultural proxy is coarse and limited by available metadata, as country-named subreddits select a particular cultural slice that engages in online discourse <ref:2606.02493#pg19>. Second, LLM judges remain imperfect on subjective annotation tasks even after human validation <ref:2606.02493#pg19>. Finally, Grice’s maxims are themselves calibrated to Western communicative norms, and observed maxim-violation rates may conflate model behavior with the metric’s own cultural assumptions <ref:2606.02493#pg19>.

Ethical Considerations

The dataset is curated solely from the publicly available Reddit Crawl, and during the study, the user ID was not used for data filtering or as metadata for response characterization <ref:2606.02493#pg19>. The subreddits employed in the study were manually filtered for NSFW content at both the subreddit and post levels <ref:2606.02493#pg19>. Human evaluators were provided sufficient time to conduct the annotations so as not to overburden them, and the LLMs employed are open-weight models allowing for reproducibility <ref:2606.02493#pg19>.

Acknowledgements

The work is supported in part by the Pioneer Center for AI, DNRF grant number P1, as well as the Carlsberg Foundation under grant agreement CFF22-1461 <ref:2606.02493#pg19>.

References

The references listed include works on cultural awareness in LLMs, anthropomorphism in LLMs, and conversational maxims in NLP <ref:2606.02493#pg19>. The study builds upon prior work examining insider/outsider positioning and the role of cultural context in language models <ref:2606.02493#pg19>. The framework is designed to provide a diagnostic lens for identifying framing divergences via a multidimensional evaluation <ref:2606.02493#pg19>. The analysis utilizes Generalized Estimating Equations (GEE) to yield population-level estimates while accounting for the non-independence of responses from three LLMs to the same post <ref:2606.02493#pg19>. The findings invite investigation into how training variations shape models’ default response tendencies <ref:2606.02493#pg19>. The paper calls for caution regarding the use of excessive anthropomorphic cues, as this can be risky <ref:2606.02493#pg19>. The work provides a starting point for evaluations that are not content-focused benchmarks such as CultureBank or BLEnD <ref:2606.02493#pg19>. The dataset supports examination at a granular 57-subreddit level of how different communities employ these response characteristics <ref:2606.02493#pg19>. The results from FRANZ invite investigation into how training variations shape models’ default response tendencies <ref:2606.02493#pg19>. The paper provides a starting point for evaluations that are not content-focused benchmarks such as CultureBank or BLEnD <ref:2606.02493#pg19>. The analysis utilizes Generalized Estimating Equations (GEE) to yield population-level estimates while accounting for the non-independence of responses from three LLMs to the same post <ref:2606.02493#pg19>. The findings invite investigation into how training variations shape models’ default response tendencies <ref:2606.02493#pg19>. The paper provides a starting point for evaluations that are not content-focused benchmarks such as CultureBank or BLEnD <ref:2606.02493#pg19>. The analysis utilizes Generalized Estimating Equations (GEE) to yield population-level estimates while accounting for the non-independence of responses from three LLMs to the same post <ref:2606.02493#pg19>. The findings invite investigation into how training variations shape models’ default response tendencies <ref:2606.02493#pg19>. The paper provides a starting point for evaluations that are not content-focused benchmarks such as CultureBank or BLEnD <ref:2606.02493#pg19>.

Improvements for AI systems

Implement FRANZ for automated response characterization to conduct a communicative audit of LLM responses along four dimensions: cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims. This allows developers and auditors to provide a diagnostic lens to identify when response framing diverges via a multidimensional evaluation.
Develop model-specific diagnostic reports based on the observed frequency differences across response characteristics. The paper observes that Mistral shows the strongest tendency to employ insider positioning as well as generalizing language; Gemma displays the best adherence to maxims; and Llama appears most constrained, which can be used for systematic model-level tendencies analysis.
Create a co-occurrence heat map visualization based on Generalized Estimating Equations (GEE) to show how response characteristics interact across countries and categories. This helps identify patterns such as the coupling observed in Figure 3, where insider framing seems to be increasing the likelihood that a response will be judged as empathetic, validating, and emotional.
Integrate a mechanism for caution regarding bidirectional coupling between cultural positioning and anthropomorphism in model steering. Since extraneous use of one can inflate the other, this allows for more nuanced optimization strategies during model debiasing efforts to avoid distorting both automated and human judgments.
Establish a framework to assess adherence to Gricean maxims as an independent signal orthogonal to cultural positioning, generalizing language, and anthropomorphism. This reveals that a response can adhere well to communication norms while still being outsiderpositioned or employing generalizing language, informing multi-objective optimization challenges for model alignment.

Abstract

Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions, where users are sensitive to how responses are communicated, not just whether the answers are correct. Existing LLM evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed. To this end, we introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions: cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims. To enable this evaluation, we contribute SQUARE - a corpus of 376k subjective questions sourced from 57 subreddits, and mapped to 7 countries and 19 question categories. We demonstrate FRANZ's applicability by scoring responses from three open-weight LLMs. We observe that LLMs show statistically significant differences in the frequency with which they employ each response characteristic. Unlike single-dimensional audits, FRANZ reveals that insider positioning and anthropomorphism are positively coupled, with the degree of coupling varying by country, providing a diagnostic lens for identifying framing divergences.