RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

arXiv:2608.26832 · cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Comments: Accepted by EMNLP 2026 Findings

Code: https://github.com/openai/tiktoken

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios.

Terminology

Abstract

Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.

Sources

Related papers