Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions

arXiv:2610.08129 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions".

Jane: The gist The results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner environment.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title itself suggests they are moving beyond just "knowing" what the partner is saying to actually seeing if that knowledge translates into doing something useful in a cooperative setting. They set up a controlled environment using Hanabi-derived settings where agents have different communication conventions, and the same hint can mean different things depending on who sent it.

Jane: So, they’re testing two key aspects: whether the model can decode what the partner is communicating, and whether that decoding actually results in cooperative decisions. It sounds like a very nuanced test of agency.

Lu: They use linear probes to see how well different parts of the model recover those conventions—specifically distinguishing between intent versus target information when looking at hidden representations. This gives them a way to measure the "knowing" part, as they call it in their setup.

Meng: But the paper immediately points out a problem: even when they can decode that convention information, receiving decisions don't consistently follow the sender’s convention, which means decoding doesn't automatically mean acting correctly. That’s a critical failure point for real-world application.

Lalam: I agree with Meng; it shows that the gap between understanding a signal and applying it to an action is still quite wide, even when the model has read the hidden states of the interaction history.

The paper's summary: Tom: The summary boils down to comparing linear probes against action translation interventions. They found that probing for partner convention information is much easier on the intent axis than on the target axis when looking at hidden representations, but receiving decisions don’t consistently follow the sender’s convention even if that information is available.

Jane: So, they’re saying the model gets a better read on what someone *wants* to say than what they *want* to do next, and that distinction is key. That makes sense because intent and action are different things in communication.

Lu: They then compare two ways of presenting this information: presenting it as a rule, or translating it directly into an action recommendation. The results show that rule statements only yield modest changes in cooperation compared to what happens when they translate the convention into a direct action suggestion.

Meng: That’s useful for deployment because if we can give the model a translation rather than just telling it "follow this rule," the performance jump is bigger on average, which suggests translating information is more effective than just reading it as a static rule.

Lalam: It reinforces what we've seen: action recommendations seem to have much stronger responses when compared to stated rules, which makes sense because we’re focusing on the actual output decisions.

The paper's improvements: Tom: The paper suggests a few ways to improve this. First, they look at activation transfers from oracle actions and non-oracle hint restatements as donors; they found that these donors actually improved intent accuracy on both action classes. That’s a clear mechanism for boosting the model’s ability to get the right intention.

Jane: An oracle action donor—basically feeding the model what the correct decision was—seems to give it a significant boost in understanding its goal, which is helpful because we know intent accuracy is important for cooperation.

Lu: They also found that combining "the hint type and the predicted convention" provides a much more reliable decoding than just looking at the resulting intent decision alone, reaching an accuracy of zero point eight one when they combine those two readouts outside the model.

Meng: Combining those two readouts sounds like a good way to refine the internal representation because it’s better than relying on just one piece of information, which is something we need when training systems to be robust.

Lalam: That combination suggests that while decoding the convention is possible, applying it effectively requires more than just reading the hidden state; external processing seems necessary for a reliable output.

Conclusion: Tom: So, to wrap up, they’re distinguishing between how well the model reads conventions, how sensitive it is to that information, and its actual cooperative performance by comparing linear probes against action translations in an unfamiliar partner environment. They found that convention readout, behavioral adaptation, and game score need separate evaluation to fully understand this.

Jane: It seems the main thing they’re saying is that simply having the internal representation of what the partner is doing isn't enough; you also have to figure out how much it actually pushes your final decision in line with their conventions.

Lu: The implication for research is that we can isolate these components and test them separately, rather than just looking at a single score like a game win rate.

Meng: From an engineering standpoint, this means we can target specific weaknesses—like the translation step or the intent accuracy—instead of trying to fix everything at once with one big fine-tuning run.

Lalam: I think it suggests that for future work, we need to focus on interventions that actively force the decision axis to align with what the partner is communicating rather than just hoping the underlying representation is enough.

Yuhwan Jeong, Jinnyeong Yang, Kuk-Jin Yoon

KAIST

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 83/100

The gist: The gist The results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner

Key concepts

Convention Decodability: This refers to how well an LLM can understand the hidden communication rules or conventions used by a partner it hasn't seen before. The research found that LLMs are significantly better at decoding the 'intent' behind these conventions than the actual 'target' outcomes, suggesting they grasp what the partner *wants* to communicate more easily.
Sensitivity to Convention Information: This measures how much an LLM's behavior changes based on whether it receives information about a partner's convention. The study showed that when presented with action recommendations rather than general rules, the model was much more sensitive and responsive to that specific information, indicating a strong link between actionable advice and behavioral adaptation.
Action Translation vs. Rule Statements: This compares two ways of presenting partner information: as a general rule (e.g., 'always do X') or as an external action recommendation (e.g., 'do Y'). The results showed that translating the convention into a specific action recommendation led to larger average improvements in cooperative performance compared to simply stating the convention as a rule.
Cooperative Performance: This is measured by how successfully LLMs adapt their receiving decisions to align with their partner's hidden conventions during interactions. The study concluded that while intent decoding is possible, reliably translating that decoded information into consistent cooperative actions remains a significant challenge for LLMs.

Summary

The gist The results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner environment.

How it works

The study investigates how Large Language Models (LLMs) use interaction history to adapt their behavior without updating the base model’s weights. This is done in a controlled Hanabi-derived environment where agents have different communication conventions, and the same hint can refer to different cards or request different decisions depending on the sender’s conventionThe same hint can refer to different cards or request different decisions depending on the sender’s convention.

The researchers developed a controlled cooperation setting that separates sender-specific convention inference, interpretation of the current hint, and receiving decisions by using scripted interactions where agents retain their own sending convention while receiving hints from a partner whose convention is not explicitly disclosedWe develop a controlled cooperation setting that separates sender-specific convention inference, interpretation of the current hint, and receiving decisions by using scripted interactions where agents retain their own sending convention while receiving hints from a partner whose convention is not explicitly disclosed.

Key Findings on Convention Decoding

The experiments show that partner-convention information is more readily decoded on the intent axis than on the target axis when probing hidden representationsOur experiments show that partner-convention information is more readily decoded on the intent axis than on the target axis when probing hidden representations.

However, receiving decisions do not consistently follow the sender’s convention even when information is availableHowever, receiving decisions do not consistently follow the sender’s convention even when information is available.

Impact of Information Presentation

The comparison between presenting convention information as a rule versus translating it into an action recommendation yielded different results regarding cooperationRule statements yield modest and model-dependent changes in cooperation, whereas action translation produces larger gains on average.

Specifically, Matched-state analyses show much stronger responses to action recommendations than to rule statements when comparing probe-predicted and ground-truth conventionsMatched-state analyses show much stronger responses to action recommendations than to rule statements when comparing probe-predicted and ground-truth conventions.

Intervention Effectiveness

Activation transfers from oracle action and non-oracle hint restatement donors improved intent accuracy on both action classesActivation transfers from oracle action and non-oracle hint restatement donors improved intent accuracy on both action classes.

The results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner environmentThe results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner environment.

Limitations

The study highlights limitations in turning available partner information into receiving decisionsThe study highlights limitations in turning available partner information into receiving decisions.

In conclusion, Convention readout, behavioral adaptation, and game score therefore require separate evaluation to understand the full picture of LLM cooperationIn conclusion, convention readout, behavioral adaptation, and game score therefore require separate evaluation to understand the full picture of LLM cooperation.

How it works

The study investigates how Large Language Models (LLMs) use interaction history to adapt their behavior without updating the base model’s weightsThe study investigates how Large Language Models (LLMs) use interaction history to adapt their behavior without updating the base model’s weights.

Improvements for AI systems

Improved partner understanding through action translation: Implement an intervention that translates convention information into current decisions, as this yield[s] larger gains on average compared to rule statements, demonstrated by matched-state analyses show much stronger responses to action recommendations than to rule statements.
Enhanced intent accuracy via oracle action transfer: Use a donor run that provides the target slot and intent decision obtained by applying the true convention, as this intervention can raise intent accuracy significantly, with one study showing an improvement from 0.603 to 0.694 for Qwen3-8B.
Convention decoding refinement: Develop a system that combines the hint type and the predicted convention more reliably than the resulting intent decision, as this combination reaches an accuracy of 0.81 when combining the two readouts outside the model, suggesting that convention information is decodable but its application requires external processing.
Adaptive rule application learning: Fine-tune a LoRA adapter on receiving decisions where the adapter applies stated rules almost perfectly, allowing the system to learn the rule-application step itself, thereby improving performance beyond what is achievable by simply recovering the convention.
Decision-specific intervention strategy: Apply interventions based on the decision axis, as target choices improve with a corrected convention, whereas intent choices respond to its translation into the current target and intent decisions, ensuring that improvements are targeted rather than merely shifting default action tendencies.

Abstract

Cooperation with unfamiliar partners requires adapting to communication conventions that are not known in advance. We study this problem in a controlled Hanabi-derived environment with scripted hint generation, LLM-controlled receiving decisions, and frozen model weights. Across eight LLMs, linear probes recover intent conventions substantially more accurately than target conventions, yet receiving choices do not consistently agree with the sender's convention. We compare probe-predicted and ground-truth conventions presented either as general rules or as externally computed action recommendations. Rule statements yield modest and model-dependent changes in cooperation, whereas action translation produces larger gains on average. In a Qwen3-8B case study, matched-state statement reversals reveal much greater sensitivity to action recommendations than to rule statements. Activation transfers from oracle-action and non-oracle hint-restatement donors improve intent accuracy on both action classes, but the tested alternatives do not reliably reproduce these benefits. Together, these results distinguish convention decodability, sensitivity to convention information, and cooperative performance, and highlight limitations in turning available partner information into receiving decisions.