The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction

arXiv:2510.06198 · cs.CL, cs.IR · Submitted 2025-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The Answer Lies Within".

Jane: Here's the gist of the paper. The gist The framework introduces COGRE, a cognitively-inspired reasoning structure, and HIT@DICT, an intermediate reinforcement learning reward strategy to enable explainable relation extraction in large language models by addressing attention misalignment and granularity mismatches.

Jane: COGRE mimics human text processing through three steps: (1) chunking from text into logical propositions; (2) anchoring certain phrases as cues; and finally (3) integrating these cues through a verbalized explanation The Construct-Integration model describes comprehension in four steps: forming concepts, elaborating, inferring new propositions, and integrating them into a representation (Kintsch, 1988).

Paper summary: Tom: So wrapping up this talk on "The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction," we’ve seen how COGRE provides the structure, and HIT@DICT provides the smart way to train it by using a self-derived reward dictionary.

Jane: The implication is that for models like these large language models, if you want them to do tasks requiring reasoning without pre-labeled data, focusing on *how* they reason—the process itself—and rewarding that alignment with human expectations can make a big difference in how reliable their answers and explanations are.

Lu: They’ve shown that this framework improves both accuracy and explanation quality compared to prior methods. The work suggests that by mimicking the construction-integration process of human cognition, we can solve the problem of models being misled by irrelevant tokens or missing the right level of abstraction.

Meng: From a deployment standpoint, if we can get explanations that are more concise and better aligned with what humans expect, it makes debugging those models way easier when they start breaking at scale.

Jane: It really moves the goalposts for how we train these systems—instead of just optimizing for the final answer score, we’re incentivizing them to show us *how* they got there in a way that makes sense to a person reading their output.

Conclusion: Tom: So we've seen how COGRE sets up the reasoning structure and HIT@DICT gives it a smart way to train itself using a dictionary based on correct answers.

Jane: Right, so what does that actually mean for these big language models? It sounds like they’re not just getting better at guessing the right answer anymore, but learning *how* to explain why they picked it in a way humans understand.

Lu: The authors are focusing on closing the gap between what the AI thinks and what a human annotator expects regarding the level of detail in those explanations.

Meng: From where I'm sitting, if you can get an explanation that is more concise but still accurate, that helps immensely when we try to put these models into actual production systems where we need fast reasoning.

Tom: Exactly. They’re tackling those two problems: the AI getting distracted by nonsense words instead of the actual connection between the things it's looking at, and it not talking at the right level of detail for what a person would actually read.

Jane: It sounds like they took something inspired by how humans process text—like chunking and anchoring—and layered reinforcement learning on top to make sure those explanations match what we need.

Lu: The paper shows that this combination doesn't just boost the final score; it actually improves the quality of the explanation itself, which is a big deal for trust.

Meng: And they even showed that by using their own AI to build that reward dictionary from correct answers, you get a pretty consistent set of keywords. That’s good because it means the system is learning from its own successful reasoning attempts.

Tom: So the title "The Answer Lies Within" really speaks to how this framework makes the model's *internal* process—the reasoning—the source of its reliable output, not just some external training trick.

Jane: It suggests that for complex tasks without clear labels, we don't need to give the AI every single answer; we just need to teach it the right way to think about the problem.

Lu: The implication is that this kind of self-derived reward mechanism could be a key ingredient in making models more robust when they encounter new kinds of reasoning problems they haven't seen before.

Meng: It’s definitely interesting because it shifts the focus from just training on data to training on *reasoning*, which is what we need for real-world use.

Xinyu Guo, Zhengliang Shi, Minglai Yang, Mihai Surdeanu

University of Arizona · Shandong University

cs.CL, cs.IR

Submitted: 2025-10-07

Updated: 2026-10-21

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The framework introduces COGRE, a cognitively-inspired reasoning structure, and HIT@DICT, an intermediate reinforcement learning reward strategy to enable explainable relation extraction in

Key concepts

COGRE: A cognitively-inspired reasoning framework that structures Relation Extraction (RE) into three steps: Proposition Chunking, Phrase Anchoring, and Integrative Reasoning. It mimics how humans process text by breaking down complex input into manageable propositions before integrating them.
HIT@DICT: An intermediate reinforcement learning reward strategy that approximates supervision by counting relation-relevant phrases against a credit dictionary. This dictionary is built from successful predictions, incentivizing the model to generate reasoning aligned with true relational labels.
Proposition Chunking: The first step of COGRE where each sentence is summarized into a relational proposition. This ensures reasoning starts with compressed propositions rather than processing long token sequences, mimicking how humans distill information.
Phrase Anchoring: A component of COGRE where certain phrases are anchored as cues during reasoning. This helps the model focus on specific, relation-conveying semantic elements instead of being misled by irrelevant tokens.

Terminology

Abstraction — 1× in this paper · explained in 1 episode
Accuracy — 1× in this paper · explained in 1 episode
Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist The framework introduces COGRE, a cognitively-inspired reasoning structure, and HIT@DICT, an intermediate reinforcement learning reward strategy to enable explainable relation extraction in large language models by addressing attention misalignment and granularity mismatches.

How it works

COGRE mimics human text processing through three steps: (1) chunking from text into logical propositions; (2) anchoring certain phrases as cues; and finally (3) integrating these cues through a verbalized explanation The Construct-Integration model describes comprehension in four steps: forming concepts, elaborating, inferring new propositions, and integrating them into a representation (Kintsch, 1988).
HIT@DICT is an intermediate reward strategy for reinforcement learning that approximates reasoning supervision by counting occurrences of relation-relevant phrases against a credit dictionary. This dictionary is built by extracting relational cues from explanations of true-positive samples generated by vanilla LLMs.
The final reward function is formulated as R = I(RAcc ≥ 1) · RHit@Dict + RAcc (1). This ensures the model is incentivized to generate correct predictions and explanations aligned well with relational knowledge.

COGRE Framework Details

The COGRE framework structures Relation Extraction (RE) into three steps: Proposition Chunking, Phrase Anchoring, and Integrative Reasoning.

** Proposition Chunking involves summarizing each sentence into a relational proposition to ensure reasoning starts with compressed propositions rather than long token sequences. 3.1 Cognitive-Structured RE describes this as formulating RE reasoning into three steps, including chunking, anchoring, and integrative reasoning. 3.2 Reinforcement Learning with HIT@DICT discusses the motivation for this framework by mimicking how humans process complex textual input. The COGRE Prompt provides a detailed guide for this process, including Summarization, Step 1 (summarize relations between subject and object), Step 2 (summarize relations between test sentence subject and object), and Step 3 (are the relations similar?). The COGRE-w/o chunking Prompt is provided in Appendix A.10.1. The COGRE-w/o reasoning Prompt is provided in Appendix A.10.2. The COGRE-w/o phrases Prompt is provided in Appendix A.10.3. Ablation studies show that all three steps contribute to the final performance, with removing any step causing a clear performance drop ranging from –1.66% to –21.51%. 3.3 Error Analysis of LLM on RE-reasoning analyzes common failure patterns, such as failing to focus on semantics that truly convey the relation or failing to align with the granularity of the RE human-annotation schema. Cases 1 through 6 illustrate these failures across different stages and reward settings, demonstrating how HIT@DICT improves alignment in some cases, for example, Case 3 where it changes the relation from non-similar to similar. The human evaluation results show that models trained with the HIT@DICT reward consistently outperform both untrained models and accuracy-reward–trained models. 3.4 Ablation Study analyzes the effectiveness of each component, revealing that phrase anchoring mainly improves precision, while chunking and reasoning improve both precision and recall, with a larger impact on recall. The results show that models trained with HIT@DICT tend to include relational cues aligned with gold labels in their explanation. 3.5 Hyperparameter Sensitivity Analysis shows that the model’s performance remains stable across various hyperparameter configurations, indicating robustness to changes in settings. The F1 score remains largely consistent at 48.12 ± 0.81. The training dynamics analysis shows that HIT@DICT accelerates the convergence of training and provides more stable training, with Qwen2.5-14B-Instruct showing steady growth compared to Only Acc. HIT@DICT also encourages more concise explanations, compressing response length to 75–90 tokens. The human evaluation confirms that HIT@DICT combined with accuracy reward enables more concise summaries and better alignment with human annotations. 3.6 Error Analysis of LLM on RE-reasoning identifies two common failure patterns: failing to focus on semantics that truly convey the relation and failing to align with the granularity of the RE human-annotation schema. The COGRE framework is specifically designed to mitigate these pitfalls. 3.7 Comparison of Extracted Dictionaries by Open-source and Closed-source LLMs shows that the two models produce highly consistent keyword sets, with an overall overlap of 0.91954 for the NYT29 training set. The performance of the method with dictionaries built from different LLMs indicates robustness to dictionary replacement. The human evaluation confirms that our reward design encourages models to generate more concise and label-aligned reasoning. 3.8 Error Analysis of LLM on RE-reasoning identifies two common failure patterns: failing to focus on semantics that truly convey the relation and failing to align with the granularity of the RE human-annotation schema. The COGRE framework is specifically designed to mitigate these pitfalls. 3.9 The paper concludes that our framework achieves enhanced accuracy and explanation quality, and human analysis confirms that our reward design encourages models to generate more concise and label-aligned reasoning. The framework is loosely inspired by cognitive science, decomposing RE into three humanlike text-processing steps. 4. Limitations This work does not investigate the internal causal mechanisms of LLMs, but instead focuses on improving their textual reasoning process. Ethical Considerations state that this work does not involve human subjects research beyond standard human evaluations. The authors confirm that all technical ideas, experiments, analysis, and final verification were conducted by the authors. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026 The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026. The paper is available as arXiv:2510.06198v3 [cs.CL] 13 Jun 2026 <ref:

Improvements for AI systems

COGRE framework will structure Relation Extraction into three steps mimicking human text processing: chunking from text into logical propositions; anchoring certain phrases as cues; and finally (3) integrating these cues through a verbalized explanation. This bottom-up design mitigates LLM attention misalignment by forcing the model to construct a coherent logical chain rather than relying on superficial lexical overlap.
HIT@DICT reinforcement learning will provide fine-grained supervision for reasoning by rewarding relation-relevant phrases in reasoning derived from a self-generated credit dictionary, which is built from explanations of true-positive samples generated by vanilla LLMs. This strategy encourages models to align their reasoning with relational labels, as shown by the improvement of +23.46% points over baseline performance.
The dual evaluation protocol will assess both RE accuracy and explanation quality using a rubric that awards points for a correct and concise summarization of the support sentence, a correct and concise summarization of the test sentence, and an explanation that is logically reasonable. This allows researchers to gauge if the model's generated explanations are closely aligned with gold labels.