Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory

arXiv:2605.19952 · cs.CL · Submitted 2026-05-19 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Rethinking How to Remember".

Jane: Here's the gist of the paper. The gist The proposed TriMem architecture maintains three coexisting representation granularities—raw dialogue segments, extracted atomic facts, and synthesized profiles—to overcome the limitations of existing fact-centric memory systems by ensuring storage fidelity, efficient retrieval, and deep reasoning capabilities.

Tom: How it works 1.

Jane: TriMem establishes a three-level architecture leveraging raw dialogue to guarantee information fidelity in storage The system builds source dialogue identifiers to preserve pointers to original dialogue segments for each compressed memory entry.

Tom: 2.

Paper summary: Tom: Looking back at "Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory", the main point is moving past just using extracted atomic facts as the single unit for storage, retrieval, and reasoning.

Jane: They propose TriMem which uses raw dialogue segments for fidelity, atomic facts for efficient retrieval, and synthesized profiles to build a deep understanding during reasoning.

Lu: It’s about keeping those three levels of representation going at once so you don't lose the detail or the ability to reason deeply over scattered pieces of information.

Meng: From an engineering standpoint, it sounds like a way to get better reliability without having to constantly retrain the entire model just for memory improvements.

Tom: The authors are using TextGrad-based prompt optimization which allows them to iteratively refine those extraction and profiling prompts based on response quality feedback, achieving lifelong evolution without parameter updates.

Jane: So the implication is that we can build agents that are more robust over time because their memory system is constantly getting smarter through those prompt adjustments.

Tom: It’s a big shift from just relying on a fixed set of extracted facts to having a dynamic, evolving memory architecture that supports complex reasoning.

Lu: They show this structure works across different model sizes and structures in their experimental validation on LoCoMo and PerLTQA benchmarks.

Meng: If we can deploy something like this that handles the scale and complexity better, it’s a step toward making agent interactions genuinely reliable in production environments where conversations are long.

Conclusion: Tom: So we're wrapping up this look at "Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory." The core idea is that you can't just rely on these small, extracted facts anymore.

Jane: Right. It seems like they’ve built a memory system with three levels of detail, not just one simple fact storage.

Lu: Exactly. They’re talking about raw dialogue segments for the original fidelity, then atomic facts for fast retrieval, and finally synthesized profiles to get that deep reasoning going.

Meng: From my side I'm wondering how they manage all those different scales simultaneously without it becoming a nightmare to run at scale.

Lalam: Based on what I’ve processed, the authors are focusing on making this system adaptable over time through prompt optimization, not just static rules.

Tom: That's the TextGrad part, right? They’re treating the prompts themselves as something that learns and gets refined based on how well the AI answers questions.

Jane: So it's not just about extracting facts once and forgetting them; it’s about a memory system that keeps evolving as it interacts with more data.

Tom: It means we move away from this old way of thinking where you just compress everything into these little chunks, because that loses too much context.

Lu: That lossy compression is a big issue because you lose the fine-grained details of what actually happened in the original conversation.

Meng: I think for me the practical thing is how they handle that on real traffic. Does this complex structure just slow down inference too much?

Jane: The experiments they ran on LoCoMo and PerLTQA show that it balances storage fidelity and reasoning performance really well across different model sizes.

Tom: So, the authors are claiming this approach consistently beats strong memory baselines in terms of both accuracy and efficiency.

Lalam: I’m confident in that claim because the structure they designed—the way they group facts into person-specific profiles—directly addresses those shallow reasoning problems we talked about earlier.

Jane: It suggests that for long term agent interactions, having a memory system that understands relationships and context deeply is much more reliable than just looking up isolated pieces of information.

Tom: It definitely shifts the focus from just getting the right answer in one shot to building an agent with persistent, multi-faceted knowledge over time.

Lu: And their future work points toward building even higher order structures on top of these profiles, like event chains or social networks for even deeper understanding.

Jane: That sounds like a really interesting direction for making agents truly robust in complex scenarios.

Jingwei Sun, Jianing Zhu, Jiangchao Yao, Tongliang Liu, Bo Han

Hong Kong Baptist University · The University of Texas at Austin · Shanghai Jiao Tong University · Sydney AI Center, The University of Sydney

cs.CL

Submitted: 2026-05-19

Updated: 2026-10-20

Comments: Accepted by NeurIPS 2026

Project page: https://tmlr-trimem.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The proposed TriMem architecture maintains three coexisting representation granularities—raw dialogue segments, extracted atomic facts, and synthesized profiles—to overcome the

Key concepts

Raw Dialogue Segments: These are the original pieces of conversation kept in memory exactly as they were spoken. They are stored with source identifiers to ensure perfect storage fidelity, preventing the loss of fine-grained details that occurs when information is compressed into facts.
Extracted Atomic Facts: These are structured, simplified pieces of information derived from dialogue, such as entities or events. They serve as a compact format for efficient memory retrieval and matching against queries, but they lack the context of the original dialogue.
Synthesized Profiles: These profiles aggregate scattered facts related to a specific entity (like a person) into a holistic semantic understanding. Instead of just knowing isolated facts, the profile captures complex relationships and behavioral tendencies necessary for deep reasoning.

Terminology

Fidelity — 1× in this paper · explained in 2 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes

Summary

The gist The proposed TriMem architecture maintains three coexisting representation granularities—raw dialogue segments, extracted atomic facts, and synthesized profiles—to overcome the limitations of existing fact-centric memory systems by ensuring storage fidelity, efficient retrieval, and deep reasoning capabilities.

How it works

TriMem establishes a three-level architecture leveraging raw dialogue to guarantee information fidelity in storage The system builds source dialogue identifiers to preserve pointers to original dialogue segments for each compressed memory entry.
It utilizes extracted atomic facts for efficient memory retrieval, relying on key facts to enable retrieval and then matching against these facts.
Synthesized profiles aggregate dispersed facts into holistic semantic understanding for deep reasoning, providing in-depth understanding over the facts to ensure the reliability of reasoning.
The system adopts TextGrad-based prompt optimization, which iteratively refines extraction and profiling prompts via response quality feedback, achieving lifelong evolution without any parameter updating.

Limitations of Existing Systems

Existing fact-centric designs suffer from three critical limitations: lossy storage, shallow reasoning, and suboptimal prompts under heterogeneous dialogue styles.

Lossy Storage occurs because the fact extraction process inevitably introduces lossy compression of raw information, where fine-grained details are discarded. For example, the modifier trans in the original dialogue is omitted during fact extraction.

Shallow Reasoning arises because reasoning mechanisms solely relying on extracted facts completely lack the ability of deep comprehension towards entity semantic portraits or behavioral tendency modeling. This means reasoning performance on multi-evidence questions is considerably inferior to that on single-evidence ones.

Suboptimal Prompts result from unstable extraction granularity with fixed prompts, as static hand-written extraction prompts fail to adaptively accommodate heterogeneous dialogue content, making it impossible to maintain consistent and rational fact extraction granularity.

TriMem Architecture Details

The framework consists of three core components: (1) Given a set of dialogue, we extract factual information and bind each fact with raw dialogue identifier and profile identifier to implement memory construction. (2) Given a question, the agent first analyzes the required information and keywords to generate search query. It then matches with extracted facts to retrieve relevant memory, and further obtains raw dialogues with full details and integrated profiles via predefined index to complete memory retrieval. (3) According to the reasoning performance, we invoke a powerful senior model to conduct failure analysis and provide revision suggestions for system prompts.

Memory Construction Pipeline

The memory construction pipeline involves several steps:

Dialogue Partitioning: The raw dialogue H is partitioned into a sequence of overlapping windows W = fseg(H; l, s) = wi where the overlapping design ensures that contextually related utterances spanning window boundaries are not artificially severed.
Fact Extraction: For each window wi, an agent φ driven by extraction prompt pe extracts structured tuples ei = f(i)restate, f(i)time, f(i)person, etc., where each dimension corresponds to a dedicated extraction function.
Source Dialogue Identification: A crucial addition is the additional dimension fsrc to the extraction prompt to obtain a source dialogue identifier ei.src ≜ [rt, ut ∈ wi] for each entry ei to address the lossy storage problem.

Scattered Fact Integration and Reasoning

To address shallow reasoning, structured profiles are constructed over scattered facts:

Grouping Facts: Extracted facts are grouped according to different person p ∈ V into Ep = [ei ∈ E f(i)person = p], ∀p ∈ V.
Profile Synthesis: Driven by a profile prompt pψ, the person-specific entries are integrated and synthesized into a structured profile Prof(p) = φ(Ep,pψ), which captures multi-faceted entity-level understanding. This allows agents to obtain integrated profiles during reasoning through the person identifier.

Lifelong Evolution via Prompt Optimization

The system employs TextGrad (Yuksekgonul et al., 2024) to optimize prompts, treating them as joint trainable parameters θ = (pe,pψ) ∈ Σ∗ ×Σ∗. The complete optimization objective is θ∗ = argmin θ E(q,a∗), where ScoreLLM evaluates answer quality via an LLM judge. TextGrad backpropagates the loss through natural language editing suggestions given by LLM agent TextGradLLM ge,gψ = ∂L/∂θ = TextGradLLM(L, θ) ∈ Σ∗. The prompts are then updated by applying these textual gradients as rewriting instructions p(t+1)e = p(t)e ⊕ g(t)e, and p(t+1)ψ = p(t)ψ ⊕ g(t)ψ.

Experimental Validation

Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong memory baselines. The performance comparison shows that TriMem achieves superior performance-efficiency balance with representative efficient models. Ablation studies confirm the necessity of entity profiles and raw dialogues for reasoning reliability and storage fidelity. The system is compatible with models of various parameter sizes, verifying its extensive superiority and broad applicability.

Conclusion

TriMem proposes a three-level architecture that maintains coexisting representation granularities—verbatim dialogues for storage fidelity, atomic facts for retrieval efficiency and progressively synthesized profiles for deep reasoning. This design effectively eliminates the adverse effects of lossy compression and isolated factual pieces. The research highlights a crucial yet long-overlooked issue in existing agent memory systems: these methods typically rely solely on extracted atomic facts as the unit across storage, retrieval, and reasoning stages. TriMem provides a new paradigm that can effectively support long-term agent interactions.

Future Work

Future exploration should focus on extending profile construction beyond person-level entities to higher-order relational structures, such as event chains, social networks, or temporal trajectories. Additionally, the focus can be placed on providing more precise and fine-grained optimization mechanisms for TextGrad to further improve the lifelong evolution capability of the memory system.

Improvements for AI systems

Establish a Tri-Granularity Memory Architecture to balance fidelity, retrieval efficiency, and reasoning quality by maintaining three coexisting representation granularities. This system allows for verbatim dialogues for storage fidelity, atomic facts for efficient memory retrieval, and synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep reasoning.
Implement Dense Storage with Efficient Retrieval by binding each extracted fact with a source dialogue identifier, addressing the lossy storage problem where fine-grained details are discarded. This ensures that even when relying on extracted facts, there is a traceable link to original dialogue segments, allowing for recovery of unstored details.
Construct Structured Profiles over Scattered Facts by grouping extracted facts according to persons and synthesizing them into profiles, which provides understanding over scattered facts to avoid shallow reasoning. This moves beyond simple fact matching to capture multi-faceted entity-level understanding, including identity attributes (e.g., gender, age, hometown), inferred personality traits.
Employ TextGrad-based Prompt Optimization to achieve lifelong evolution without any parameter updating, iteratively refining extraction and profiling prompts via response quality feedback. This allows the system to continuously evolve by treating prompts as joint trainable parameters, adapting to highly heterogeneous real-world dialogue scenarios.
Enhance Question Comprehension through Information Requirements Analysis by generating a structured plan that determines the minimal set of targeted search queries needed to gather the required information. This ensures retrieval is highly focused, moving beyond direct question matching to gathering specific key entities, events, or concepts required for comprehensive answers.
Implement Context-Aware Reasoning via Profile Integration by using profiles during reasoning to infer behavior and values rather than relying solely on retrieved facts. The system must USE WORLD KNOWLEDGE to identify described but unnamed things, such as naming a technique like the Pomodoro technique when only a description is given.
Develop Adaptive Temporal Resolution in Extraction Prompts by explicitly instructing the extractor to resolve relative time expressions into absolute dates based on dialogue timestamps, ensuring that the timestamp field should be the EVENT time, NOT the conversation time.