Jev-ToM: Evaluating Theory of Mind in Typed Decision Models
cs.CL, cs.AI
Submitted: 2026-10-13
Updated: 2026-10-13
Comments: Code and evaluation materials: https://github.com/HKUST-KnowComp/Jev-ToM
Code: https://github.com/HKUST-KnowComp/Jev-ToM
License: http://creativecommons.org/licenses/by/4.0/
The gist: Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends.
Abstract
Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.
Sources
- Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV
- Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
- MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning
- Evaluating and Benchmarking the System One Model Jev
- General Decision Models: Benchmarking and Insights Beyond Jev
- Understanding Social Reasoning in Language Models with Language Models
- JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models
- Evaluating Decision Models for Text Annotation in Computational Social Science
- JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
- PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions
- Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind
- Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It
- Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
- Mnemon: Raw Records, Fast Judgments, Slow Thoughts
- JevOut: Natural Context Can Flip Decision Models
- Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflection
- When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering