Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG

arXiv:2605.25641 · cs.CL · Submitted 2026-05-25 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Iterate Until Retrieved".

Jane: The gist Iterative Nugget Optimization (INO) is an index-time optimization method that uses production agentic RAG as a test harness to iteratively refine factual nuggets from user feedback until they…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re diving deeper into the mechanics of this Iterative Nugget Optimization for Discoverable Continual Corrections in Agentic RAG. The authors are Moshe Hazoom, Gal Patel, and Alon Talmor from Mosaic AI and The Hebrew University of Jerusalem >

Jane: Those are the folks who did the heavy lifting on this research. They’re looking at how agentic retrieval-augmented generation systems in business settings handle free-form feedback >

Lu: Their core contribution is using the production agentic RAG system itself as a test harness to iteratively refine these factual nuggets until they become discoverable and optimized for future queries >

Meng: So they're not just doing a one-time indexing job; they're building a loop that tests the nugget, probes it with queries, and reflects over failures >

Tom: Right. They start with an initial nugget based on the feedback, then they generate some test queries—a triggering query and some paraphrases—and if that nugget doesn't get used for those or probe queries, an LLM reflector steps in to revise the representation >

Jane: It’s like a quality control loop specifically for knowledge that needs to be reusable and standalone >

The paper's summary: Tom: So the summary boils down to this: they take user feedback, filter it down to actionable factual corrections, structure those into nuggets, and then run them through the production RAG stack in a loop until they become discoverable for relevant future traffic >

Jane: The key is that the nugget has to be encoded conservatively so it doesn't introduce wrong knowledge but still have enough context to be reusable and standalone >

Lu: They explicitly point out that a straightforward approach of just inheriting the wording and context from the original interaction often ends up being too narrow because future users ask from different roles or use different company vocabulary >

Meng: So they are trying to solve the problem where a correction is only useful for one specific way of asking a question, but they want it to be useful for many ways >

Tom: Precisely. The goal is to avoid the nugget becoming so broad that it gets retrieved for completely unrelated traffic, which is a big risk in large knowledge bases >

Jane: And they show how this iterative process works by creating candidate nuggets and running them through the same agentic RAG system that will serve future users >

The paper's improvements: Tom: Now let’s talk about what they actually suggest improving. They compared five different strategies for nugget construction to see which one optimizes discoverability and usage of factual corrections best >

Jane: They looked at things like a standard nugget, a trigger-query anchor, synthetic query anchors, and combinations of those >

Lu: One interesting strategy is the "Synthetic query anchors" where an LLM generates five short future-user queries that the nugget should answer and appends them to the nugget >

Meng: That seems ambitious because generating five queries for every correction adds overhead. What makes one better than another in terms of what actually helps retrieval?

Tom: The results showed that all these strategies consistently improved results across retrieval, citation, and answer feedback compliance compared to the standard nugget baseline >

Jane: But they found that using "Trigger plus synthetic anchors" seems particularly strong because it combines a paraphrase of the original query with four other queries >

Conclusion: Tom: So to wrap up on this paper, Iterative Nugget Optimization for Discoverable Continual Corrections in Agentic RAG shows that you can optimize the artifact itself rather than just relying on its initial creation >

Jane: It proves that by putting the nugget construction into an index-time optimization loop, we can ensure these factual corrections are actually discoverable and usable for future, different types of queries >

Lu: The implication is that we need a more dynamic way to think about how knowledge is indexed in RAG systems so it adapts to evolving user language without manual intervention >

Meng: From a deployment standpoint, the cost they report for this optimization process is about zero point three one per correction and under two minutes per nugget in setup which makes it relatively efficient for running at scale >

Tom: So we’ve seen how they use the production agentic RAG system as a test harness to make sure these nuggets are discoverable, and it’s a practical method that can be run offline with a small cost per correction >

Jane: It gives us a solid blueprint for handling the inevitable stream of factual corrections in complex B2B settings where knowledge bases are constantly being updated >

Mosaic AI · The Hebrew University of Jerusalem

cs.CL

Submitted: 2026-05-25

Updated: 2026-10-19

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: The gist Iterative Nugget Optimization (INO) is an index-time optimization method that uses production agentic RAG as a test harness to iteratively refine factual nuggets from user feedback until

Key concepts

Iterative Nugget Optimization (INO): INO is an optimization loop that refines factual nuggets by testing them against a production RAG system. It starts with a correction, tests if the nugget is used, and if not, uses an LLM to revise the nugget's wording until it becomes discoverable for future queries.
Factual Nugget: A factual nugget is a concise piece of information derived from user feedback that needs to be indexed. INO treats its creation as an optimization problem, aiming to make this nugget highly usable and retrievable by the RAG system.
Agentic RAG System: This is the production system used as a test harness for INO. It simulates how real users interact with the indexed information. The method uses this system to determine if a nugget is actually retrieved and used, guiding the refinement process.

Terminology

RAG — 3× in this paper · explained in 3 episodes
Harness — 1× in this paper · explained in 2 episodes
Iterative Nugget Optimization — 1× in this paper
Retrieval-Augmented Generation (RAG) — 1× in this paper · explained in 21 episodes
Trace — 1× in this paper · explained in 2 episodes

Summary

The gist

Iterative Nugget Optimization (INO) is an index-time optimization method that uses production agentic RAG as a test harness to iteratively refine factual nuggets from user feedback until they are discoverable and optimized for future queries.

How it works

The process begins by detecting actionable factual corrections from user feedback, which are then converted into an initial factual nugget.
INO treats the construction of the nugget as a measurable index-time optimization problem.
The method starts from a factual correction to create a candidate nugget and a small set of likely user queries.
This candidate is then inserted into the relevant corpus and run through the production agentic RAG system.
If a nugget is not retrieved or used for the triggering query and probe paraphrased queries, an LLM reflector inspects the failed queries, retrieved competing documents, and generated answers to revise the nugget representation.
The reflector revises the indexable nugget representation—including title, body wording, and retrieval anchors—until it becomes discoverable.
This iterative refinement continues for at most T=3 iterations.

Key Components and Strategies

The paper compares five different strategies for nugget construction to see which optimizes discoverability and usage of factual corrections. These strategies include:

** Standard nugget:**

This baseline indexes the nugget as-is.

** Trigger-query anchor:**

This variant appends a paraphrase of the original user query q0 to the nugget.

** Synthetic query anchors:**

An LLM generates five short future-user queries that the nugget should answer, which are appended to the nugget.

** Trigger plus synthetic anchors:**

This combines a paraphrase of q0 with four other queries.

Evaluation and Results

The performance of the strategies is evaluated across three levels: Retrieval, Citation, and Answer feedback compliance. INO consistently improves results over baselines in terms of discoverability and usage of factual corrections in automated and human evaluations.

** Held-out queries:**

INO leads to large gains in nugget discoverability and usage on held-out queries. For the chat agent, INO improves retrieval and citation by 25-29 points over the standard nugget baseline.

** Support-Ticket agent transfer:**

The gains on support tickets are consistent with the harder input distribution, where transcripts contain filler and different vocabulary.

Practical Implications

INO is practical because it keeps the production agent configuration fixed and makes optimization local, auditable, and reversible at the nugget level. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in the setup. This approach is currently running in production, generating hundreds of retrieved nuggets across customers weekly.

Limitations

The study notes several limitations, including reliance on a single retrieval architecture which may respond differently to different anchor types. Additionally, the method's empirical claims are English-only and depend on frontier LLMs for extraction, reflection, and judging. The study also reports results based on the actual deployed system rather than a full model ablation.

The proposed Iterative Nugget Optimization (INO) substantially improves discoverability and usage of factual corrections compared to baselines across various B2B agents. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in the setup. This approach is currently running in production, serving hundreds of nuggets every week. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in our setup. This approach is currently running in production, serving hundreds of nuggets every week. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in our setup. This approach is currently running in production, serving hundreds of nuggets every week. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in our setup. This approach is currently running in production, serving hundreds of nuggets every week. The optimization is offline with a per-correction cost of about 0.31 and under two minutes per nugget in our setup <ref:2605.

Improvements for AI systems

Bold header: Iterative Nugget Optimization (INO) for Factual Correction Feedback

This method implements a loop that tests and refines each nugget using the production agentic RAG system as a test harness, ensuring it is discoverable and optimized for future queries. This keeps the production agent configuration fixed while optimizing the retrievable artifact, which is described as doing what amortizes work across future queries about one factual correction by changing the retrievable artifact.

Bold header: Dynamic Nugget Construction Strategies

Instead of a single construction method, systems can employ strategies like Synthetic query anchors or Trigger plus synthetic anchors to generate varied test scenarios for the nugget. This allows the system to produce a richer set of candidate nuggets, such as those in Variant D, which includes both a paraphrase of the original query and four other queries.

Bold header: Actionable Feedback Filtering via LLM Classification

A dedicated LLM-based actionability extraction step filters the stream to define an actionable factual correction as feedback that provides enough concrete information to write a reusable factual statement. This ensures that only high-value, reusable knowledge is converted into nuggets, improving the signal-to-noise ratio.

Abstract

Agentic retrieval-augmented generation (RAG) systems in business-to-business settings may often receive free-form response feedback. Rather than generic feedback signals such as style, preference, or overall response quality, we focus on actionable factual corrections. We identify these instances and convert them into compact knowledge-base entries, which we call factual nuggets. We introduce Iterative Nugget Optimization (INO), an index-time optimization method that uses the production agentic RAG as a test harness: it creates an initial nugget, probes it with the triggering query and paraphrases, reflects over failed retrieval and answer traces, and revises the nugget until it is discoverable. We evaluate INO with two production B2B knowledge-assistance agents across multiple companies that use our system: a product support agent that answers questions over company-specific knowledge bases, and a support ticket agent that assists support engineers. INO consistently improves results over baselines in terms of discoverability and usage of factual corrections, in automated and human evaluations. A controlled experiment in production shows that the nuggets improve user satisfaction with responses.