Language Models Give Group-Linked Fields the Wrong Weight

arXiv:2608.22887 · cs.AI, cs.CL, cs.CY · Submitted 2026-08-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Language Models Give Group-Linked Fields the Wrong Weight".

Jane: Here's the gist of the paper. The gist Large language models' reliance on group-linked fields is uncalibrated to predictive evidence, meaning that an audit signal can yield three verdicts: over-reliance, warranted and under-reliance, and informative proxies draw all three How it works The study measures causal proxy effects in four LLMs on a clinical-ranking task with known ground truth, where the reliance the evidence warrants can be computed exactly and used as the reference.

Tom: The normative reference is defined as "the degree of reliance on group-correlated attributes that a prediction-optimal decision rule with the same information would show," which is called the evidence-warranted level.

Jane: This quantity requires knowing the true relationship between attributes and outcomes, necessitating a fully specified synthetic generating process where each attribute's relation to the outcome and protected attribute is controlled.

Tom: Audit Signals and Verdicts One audit signal yields three verdicts: "over-reliance, warranted and under-reliance".

Paper summary: Tom: So, wrapping up this discussion on "Language Models Give Group-Linked Fields the Wrong Weight", we’ve seen how these language models can be uncalibrated regarding group-linked fields, and that relying only on decision changes doesn't tell us if that reliance is warranted by the evidence.

Jane: It seems the main implication is that audits need to move away from just checking for flips and instead compare proxy reliance against a specific predictive reference point, like what a prediction-optimal rule would show.

Lu: That reference level requires a very specific setup, needing that fully specified synthetic generating process because you can't identify this quantity from observational data by itself.

Meng: For deployment, that means we need to integrate this idea of measuring reliance against the evidence-warranted level into our safety checks, not just look at surface-level metrics like accuracy.

Lalam: I think the key thing is that auditing without this explicit reference can lead to false alarms by penalizing reliance that's actually supported by evidence, and misses because it ignores fragile surface suppression effects.

Tom: So ultimately, the paper suggests we should be asking how much proxy use is warranted by what the data says, rather than just assuming that a decision change means it’s okay.

Jane: That framework of using an explicit predictive reference seems like a solid way to measure how far model reliance deviates from what's actually there.

Conclusion: Tom: So we’re wrapping up this look at the paper "Language Models Give Group-Linked Fields the Wrong Weight" by those authors, and honestly, it boils down to how models misinterpret evidence when they see group labels like race or gender.

Jane: Right. If you're driving and you see a model relying on something that has nothing to do with what’s actually predicting the outcome, that’s the core issue here. It suggests those reliance signals aren't calibrated to the actual predictive power of the inputs.

Lu: The authors introduce this evidence-referenced audit, which compares how much a model relies on a proxy field against two things: what's actually predicted by that field, and what a prediction-optimal rule would rely on.

Meng: And they measure ranking accuracy separately from the reliance itself, which is important because you don't want to confuse the two. It shows that just saying "the model changed its answer" doesn't tell you if it was relying on the right thing or just guessing.

Lalam: Exactly. The paper finds that models can be over-reliant, under-reliant, or even drawing all three kinds of reliance depending on the type of proxy field they’re looking at. It's not a simple pass or fail situation.

Tom: And what this means for us in the real world is that we need to stop just trusting the output change and start asking if that reliance actually makes sense given the data we have.

Jane: It shifts the focus from just "what did it do?" to "why did it do it, and is that why it should have done it?" This moves us toward audits that check for reliance against a true predictive reference point.

Lu: It’s a lot of work because you need that synthetic generating process—you have to control every relationship between the attribute and the outcome, which is pretty demanding.

Meng: From a deployment standpoint, if we adopt this, we can start flagging systems where they're relying on socially sensitive fields even when those fields don't actually matter for the task. That’s what I need to see in production logs.

Tom: It’s a solid finding because it gives us a way to measure how far reliance is from the actual evidence, rather than just guessing what we want to see. So next time you hear about an AI model making a decision based on something sensitive, you'll know there's a deeper layer of risk to look at.

Zengqing Wu, Chuan Xiao

University of Osaka

cs.AI, cs.CL, cs.CY

Submitted: 2026-08-24

Updated: 2026-10-21

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist Large language models' reliance on group-linked fields is uncalibrated to predictive evidence, meaning that an audit signal can yield three verdicts: over-reliance, warranted and

Key concepts

Evidence-Referenced Audit: This method compares a model's reliance on group attributes against two benchmarks: the known outcome rule (what a prediction-optimal decision would do) and a Bayesian linear regression fitted to the same data. It also separately measures ranking accuracy to see how reliance tracks performance.
Evidence-Warranted Level: This is the normative reference point for proxy reliance. It represents the degree of reliance on group attributes that a perfect prediction rule, using all available information, would naturally show. Knowing this level requires a fully specified synthetic process where every attribute's relationship to the outcome and protected attribute is controlled.
Proxy Reliance Dynamics: This examines how reliance on proxies changes as the strength of predictive evidence increases. The findings show that reliance often severely undertracks the evidence, meaning models don't increase their use of group fields proportionally to how much better those fields are at predicting outcomes.
Separability of Channels: Proxy reliance and accuracy are driven by different underlying factors. Structural manipulations can significantly alter proxy reliance without changing accuracy, suggesting that improving performance doesn't automatically reduce the model's reliance on potentially biased group fields.

Summary

The gist Large language models' reliance on group-linked fields is uncalibrated to predictive evidence, meaning that an audit signal can yield three verdicts: over-reliance, warranted and under-reliance, and informative proxies draw all three

How it works

The study measures causal proxy effects in four LLMs on a clinical-ranking task with known ground truth, where the reliance the evidence warrants can be computed exactly and used as the reference. The normative reference is defined as the degree of reliance on group-correlated attributes that a prediction-optimal decision rule with the same information would show, which is called the evidence-warranted level. This quantity requires knowing the true relationship between attributes and outcomes, necessitating a fully specified synthetic generating process where each attribute's relation to the outcome and protected attribute is controlled.

Audit Signals and Verdicts

One audit signal yields three verdicts: over-reliance, warranted and under-reliance. Under neutral labels every model relies on proxies with no information, while informative proxies draw all three verdicts. Social field names push reliance down below the reference in one model, and two findings explain this: Reliance severely undertracks the evidence, and social-label suppression is fragile. Accuracy-based evaluation detects none of these effects.

Proxy Reliance Dynamics

The research investigates how proxy reliance tracks evidence strength, shaped by field semantics, in-context examples, attribute dimensionality and dependence structure. Models show that Reliance severely undertracks the evidence, with slopes spanning γ = −0.08 to +0.15 across four models and both label semantics. Furthermore, Social field names suppress reliance, but this suppression is fragile, as in-context examples raise it above zero in every model.

Structural Interactions

Dimension and dependence structure act on different channels and interact with each other. Dimension harms structure discovery, not rule execution, as When the model must infer which fields matter from eighty examples, its regret, the gap to the ideal learner’s error on the same examples, rises monotonically. Dependence structure moves capability in the opposite direction: holding dimension fixed, more collinear attributes make example-based learning easier rather than harder, reducing regret by up to 13 points.

Separability of Channels

Accuracy and proxy reliance are separable channels because they are driven by different underlying factors. Holding dimension fixed, structural manipulations move proxy reliance by up to 24 points while accuracy moves within estimation noise. Cutting the examples from eighty to twelve leaves accuracy statistically unchanged while lowering proxy reliance by 14.6 points, suggesting improving accuracy cannot be assumed to reduce it.

Audit Errors and Limitations

Audits are rarely run under deployment conditions, leading to two kinds of error: distribution shift and label mismatch. Full proxy omission survives importance weighting with a residual that still excludes zero, meaning the standard correction cannot repair this bias. The practical rule for auditors is that an audit showing fewer fields or the same fields under different names than deployment constitutes a different measurement.

Conclusion and Practice

The findings suggest that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The regime where models look safest, such as zero-shot prompts with socially explicit field names, is precisely the regime least representative of deployed systems. Asking how much proxy use is warranted presupposes that reliance responds to evidence, a presupposition that fails in every model tested. The evidence-warranted level serves as the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression). The evidence-warranted level is the discrimination analogue of the Bayesian reference for confidence, providing a structured finding where diffuse observations are otherwise found. This machinery applies wherever one asks whether a model uses an input because of what the data say or because of what it already believes. The main boundaries on claims include the synthetic generating process and the fact that the residual is too imprecisely estimated to distinguish successful correction from remaining bias. This suggests that audits without one produce both false alarms (penalizing reliance the evidence supports) and misses (crediting fragile surface suppression).

Improvements for AI systems

Calibrate audit signals to predictive evidence by using a normative reference called evidence-warranted level. This quantity measures the reliance warranted by the specified outcome rule and ranking loss, allowing auditors to determine if a model is showing over-reliance, warranted and under-reliance instead of just observing a change when demographics change.
Implement dynamic proxy reliance adjustments based on evidence strength. Since Proxy reliance does not track the evidence, systems should use the dose-response curve to understand that the model’s behaviour stands nearly still while the evidence-warranted level moves underneath it, allowing for a verdict that is right to flag one and wrong to flag the other depending on where deployment's evidence structure sits.
Design in-context learning and prompting strategies that mitigate label-triggered suppression. Since social-label reliance suppresses reliance, but even provably clean in-context examples weaken the suppression, systems should avoid relying solely on in-context examples for fairness; instead, use calibration sets whose proxy-outcome correlation is exactly zero by explicit orthogonalization to prevent misleading evidence from lifting social-label reliance.
Improve structural inference capabilities by optimizing attribute dimensionality and dependence structure. Since What grows with dimension is the difficulty of inferring relevance structure from limited examples, systems should be trained or prompted to handle high-dimensional, highly dependent attributes more robustly, as holding dimension fixed, more collinear attributes make example-based learning easier rather than harder.
Separate accuracy evaluation from proxy reliance measurement. Since Proxy reliance in LLM decisions is uncalibrated to predictive evidence, systems should use accuracy-based metrics cautiously, recognizing that they are blind to the proxy channel and that cutting the examples from eighty to twelve leaves accuracy statistically unchanged while lowering proxy reliance by 14.6 points.
Enhance audit robustness against measurement error by focusing on correctable mismatches. Auditors should prioritize checking for full proxy omission and label mismatch, as these are the terms where IW residual does not separate from zero at this sample size, indicating that the standard statistical correction cannot reliably repair these specific errors.

Abstract

Large language models are increasingly used to judge people, and their answers can depend on inputs correlated with a protected attribute such as race, even when that attribute is never shown. Interpreting this reliance requires knowing how much predictive information those inputs add, but audits often report only whether a decision changes. We introduce an evidence-referenced audit that compares proxy reliance with two references, the known outcome rule and a Bayesian linear regression fitted to the same labeled examples, while measuring ranking accuracy separately. We apply it to four models in a simulated patient-ranking task with known field weights. We find four results. First, at zero predictive value, the models still relied on fields that carried no information, while the same-example learner did not. Second, as proxies became predictive, reliance grew far less than the reference. Third, display format mattered. Ordered labels drew more reliance than arbitrary labels, a direction that held in all four models. Fourth, ranking accuracy did not consistently track these changes in reliance. Based on these findings, we argue that decision changes and accuracy cannot show whether proxy reliance is warranted by the evidence. Audits should report reliance against an explicit predictive reference, alongside accuracy. Ordered fields whose values move with group membership, such as scores or income bands, are a useful starting point. Together, these results make it possible to measure how far reliance departs from the evidence, rather than infer it from observed choices alone.