Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG

arXiv:2607.03929 · cs.CL, cs.AI, cs.LG · Submitted 2026-07-04 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Probe, Don't Prompt".

Jane: Here's the gist of the paper. The gist: A shallow, mean-pooled, imbalance-aware probe on a small open-source model is a drop-in replacement for the prompted GPT-3.5 metadata extractor in Multi-Meta-RAG, with +10 points set-exact overall, with the margin coming from null query abstention (it matches but does not beat a substring baseline on non-null queries), no API cost, and no allow-list drift.

Tom: How it works Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata (the news source) that it extracts from each query by prompting gpt-3.5-turbo.

Jane: This proprietary, free-form extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model.

Tom: The probe is a fixed-vocabulary multi-label classifier: its output space is exactly the 49 sources, so it is structurally incapable of the allow-list drift the prompted model exhibits.

Paper summary: Tom: Alright folks, we're talking about a paper called "Probe, Don't Prompt: A Hidden-State Probe for Metadata Filtering in Multi-Meta-RAG". This paper is looking at how to filter information from a vector store based on metadata like the news source and the date.

Jane: So, what’s the main idea here? It claims that you can replace a prompt that asks an AI like GPT-three point five for every query with something smaller and local instead <ref:2607.03929#pg1>.

Lu: Exactly. The authors are showing how to use a hidden-state probe from a small open-source language model to do this filtering without needing that proprietary API call every single time, which is pretty expensive and risky because the results aren't guaranteed to stay within the sources you allow.

Meng: I’m curious about that risk. If we can’t trust the prompt, what exactly are we trusting in this hidden-state probe?

Tom: Well, the paper says this probe is trained on the hidden states of a small language model and it's designed to be deterministic, meaning it won't drift outside the set of sources you give it.

Jane: That’s a big deal because usually when you use GPT-three point five for extraction, if it gets weird in its wording, it might generate something completely unexpected that breaks your filter <ref:2607.03929#pg1>.

Lu: The methodology they used involved a few key design choices: picking a shallow layer, using mean pooling over the tokens, and training with class-imbalance-aware loss on those sources.

Meng: Mean pooling sounds like a good way to capture the overall context of the query across all its words, right? What’s in that for us at scale?

Tom: It helps because they found that mean pooling gave them the higher out-of-fold F1 score for all four models they tested. They also picked a shallow layer, specifically index one through four of the twenty-four to thirty two transformer layers.

Jane: So why shallow layers if deep layers often seem better for understanding complex things?

Lu: The paper explains that for near-lexical source identity, the shallowest layers win because the target attribute is usually present in the surface form of the query, not in deep semantics.

Meng: Okay, so this means we don't need to run a massive model through all its layers just to get our metadata filter. What’s that cost-wise?

Tom: The paper shows that a one hundred thirty-five million parameter model lands with a set-exact accuracy of ninety point nine percent overall against an eighty-eight point zero percent baseline for substring matching, and it's only about one point less accurate than the larger one point five billion parameter model, which is pretty good for something small.

Jane: That margin of improvement they found is mostly coming from null queries where GPT-three point five never abstains, meaning it always predicts *something* <ref:2607.03929#pg1,GPT-3.5 never abstains>.

Lu: And those null queries are actually why the probe beats the substring baseline on non-null queries by a point or two across all tested models.

Meng: So, if I were trying to deploy this in production, what’s the big caveat? What stops me from just copying the GPT-three point five approach and calling it a day <ref:2607.03929#pg1>?

Tom: The main limitation they mention is that this study is based on a single dataset and domain, which is news. And they also note that source identification is near-lexical, so you already have a strong string-match baseline to compare against.

Jane: That makes sense for now. But what about the future? Where does this leave us when we think about making these kinds of systems more robust for different types of information?

Lu: The authors suggest some interesting next steps, like looking into learned date greater than or less than operators, which would be useful if we want to handle temporal queries better.

Meng: And they also mention exploring a hybrid approach using a lexical match and this probe with a null gate to try and push the accuracy past the string match baseline on non-null queries. That sounds like something we could test in our own environment.

Tom: So, in short, this paper shows that you can replace an external prompt extractor with a smaller, local model probe for metadata filtering to get about ten points higher set-exact accuracy overall without the API cost or the risk of drift.

Jane: That’s a lot of concrete numbers for something that sounds like it’s just some academic deep learning experiment. It really shows how much power is in using the right representation learned by a small model for a specific task like filtering.

Conclusion: Tom: So we just looked at how they replaced that GPT-three point five prompt extractor with this little hidden-state probe for their MultiMetaRAG system, and now we’re talking about what this whole paper actually means for us outside of the lab.

Jane: It turns out the authors took a really messy part of using big language models—that relying on them to be perfectly precise for filtering—and they found a way to make it deterministic and much more affordable.

Lu: The core idea is that you don't need the massive model to *interpret* the metadata; you just need a small, specific layer from it. They trained this probe using mean pooling and a class-balanced loss on those forty-nine news sources.

Meng: And what they showed is that this tiny probe gets about ten points better set-exact accuracy overall compared to just using GPT-three point five for the filtering part, and it costs nothing in terms of API calls for every query.

Tom: That margin comes mostly from null queries, where the old system never guessed wrong because it always gave an answer, while this new method abstains when there's no source information present.

Jane: So what does this tell a person who doesn't read research? It means we can stop treating those expensive GPT calls as mandatory for every single metadata check.

Lu: It proves that you can replace a complex, external prompt with an internal, local representation of the model to solve a retrieval problem and get better results.

Meng: From my side at the startup, if we could swap out that part of our pipeline for something this small and fast, it cuts down on inference cost significantly without sacrificing accuracy in this specific filtering task.

Tom: Exactly. It’s about making those large models do the heavy lifting for understanding, and using smaller models just to do the quick yes or no check on things like source names.

Jane: It opens up a path for building retrieval systems that aren't totally dependent on constantly paying for third-party API access every time they need to filter their data store.

Tom: Yeah, and the next thing we look at is how this kind of internal representation affects the overall retrieval performance—like whether it actually helps things get found in a multi-hop question answering task.

Mykhailo Poliakov, Nadiya Shvai

National University of Kyiv-Mohyla Academy

cs.CL, cs.AI, cs.LG

Submitted: 2026-07-04

Updated: 2026-10-20

Comments: Accepted to Computer Science and Applied Mathematics

Code: https://github.com/mxpoliakov/Multi-Meta-RAG

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: The gist: A shallow, mean-pooled, imbalance-aware probe on a small open-source model is a drop-in replacement for the prompted GPT-3.5 metadata extractor in Multi-Meta-RAG, with +10 points set-exact

Key concepts

Multi-Meta-RAG: A system designed to improve question answering by filtering a vector store using metadata like news sources. It uses a retrieval pipeline where the source information helps narrow down relevant documents before answering the main question.
Hidden-State Probe: A technique where a small language model is used not for generating text, but to analyze its internal 'hidden states'—the mathematical representations of tokens—to classify metadata. This probe learns to predict which of the 49 allowed news sources a query belongs to.
Mean Pooling: A method used during the probe pipeline where all token representations for a specific layer are averaged together. This pooling creates a single, fixed representation ($h_{ ext{mean}}$) that summarizes all the information from that layer, making the subsequent classification step more robust and consistent.
Class-Balanced Loss: A specific training method used to teach the probe classifier. Since some news sources appear much less frequently than others (class imbalance), this loss function ensures the model learns equally important patterns for rare sources, leading to better performance on those less common categories.

Terminology

RAG — 4× in this paper · explained in 3 episodes
Accuracy — 1× in this paper · explained in 1 episode

Summary

The gist: A shallow, mean-pooled, imbalance-aware probe on a small open-source model is a drop-in replacement for the prompted GPT-3.5 metadata extractor in Multi-Meta-RAG, with +10 points set-exact overall, with the margin coming from null query abstention (it matches but does not beat a substring baseline on non-null queries), no API cost, and no allow-list drift.

How it works

Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata (the news source) that it extracts from each query by prompting gpt-3.5-turbo. This proprietary, free-form extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. The probe is a fixed-vocabulary multi-label classifier: its output space is exactly the 49 sources, so it is structurally incapable of the allow-list drift the prompted model exhibits.

Key Design Choices

The success of the probe relies on three specific design choices:

selecting a shallow layer, mean pooling, and class-imbalance-aware multi-label training over the long tail of sources.

The probe is a fixed-vocabulary multi-label classifier: its output space is exactly the 49 sources, so it cannot drift outside the allow-list as the prompted model does.

The probe’s output space is exactly the fixed 49-source vocabulary, so it cannot drift outside the allow-list as the prompted model does.

Probe Pipeline and Configuration

For a query tokenised into Tq tokens we run one forward pass through a small open-source model with output hidden states=True, giving hidden states h(t)l at layer l for token t. We cache, for every layer, two pooled representations, hmeanl = 1/Tq X Tq t=1 h(t)l and hlastl = h(Tq)l as a [N, nl, d] tensor. On the pooled state hl of a single chosen layer, a 49-way multi-label head scores each source c with an independent logistic unit and one global decision threshold τ, pqc = σ w⊤ c hl + bc. Sˆq = ∪sc: pqc ≥ τ, where σ is the logistic sigmoid and features are standardised per dimension. At inference the predicted set Sˆq is written back as a source.

Performance and Findings

A set-exact head-to-head on all 2556 MultiHop-RAG queries against both GPT-3.5 and a model-free substring baseline: the probe scores 90.9% overall versus 88.0% (substring) and 80.9% for GPT-3.5, with a margin that comes entirely from null queries. On non-null queries all three stay within about a point. The overall margin comes from the null queries. On non-null queries all three methods are close (Table II): GPT-3.5 is higher on comparison (98.2% vs. 94.4%), the probe is higher on temporal (84.2% vs. 81.8%), and the substring baseline (90.8%) is on par with the probe (90.6%). The selected layer is shallow in every case, index 1–4 of the 24–32 transformer layers (Table I, Fig. 2), rather than the middle layer reported as strongest for multi-hop retrieval [5] and for general downstream use [10].

Analysis of Design Factors

Layer. We sweep every hidden-state layer under both poolings with iterative-stratified 5-fold cross-validation and select the layer with the best out-of-fold micro F1. Pooling. Mean pooling gives the higher out-of-fold F1 for all four models and at nearly every depth (Fig. 2), consistent with the source name appearing anywhere in the query rather than only at its end. Class imbalance. We therefore train each logistic unit by minimising a class-balanced binary cross-entropy.

Conclusion and Limitations

A shallow, mean-pooled, imbalance-aware probe on a small open-source model is a drop-in replacement for the prompted GPT-3.5 metadata extractor in Multi-Meta-RAG [3]: +10 points set-exact overall, with the margin coming from null query abstention (it matches but does not beat a substring baseline on non-null queries), no API cost, and no allow-list drift. The broader lesson for probing-in-the-loop systems is to read the cheapest layer that suffices. Limitations. The study is a single dataset and domain (news); the source attribute is near-lexical, so a string-match baseline is already strong; and macro F1 trails micro on the rarest sources. Future work includes learned date gt/lt operators, a lexical-∪-probe hybrid with a null gate to push non-null accuracy past string match, focal loss or per-class thresholds for the rare-source macro-F1 tail, and measuring the end-to-end retrieval impact (MRR@10, Hits@k) of swapping the extractor.

Improvements for AI systems

Replace proprietary GPT-3.5 source extraction with a deterministic probe that uses the hidden states of a small open-source language model. This eliminates API costs and network latency while preventing allow-list drift, ensuring the filter stays within the fixed 49-source vocabulary.
Implement a retrieval pipeline where only the first few layers of a small open-source model are used for metadata filtering, as the source name appears verbatim in the query in 95.4% of gold query–source pairs, making shallow layers win over deeper semantic representations for near-lexical attributes.
Utilize mean pooling over token states instead of last-token pooling to score sources, because mean pooling gives the higher out-of-fold F1 for all four models and at nearly every depth. This improves accuracy when source names appear anywhere in the query rather than only at its end.
Employ a class imbalance aware multi-label training strategy, minimizing a class balanced binary cross entropy with weights inversely proportional to class frequency, to ensure the probe accurately identifies rare sources present in the long tail of sources.
Deploy a hybrid retrieval system that combines the fixed-vocabulary probe with a null gate, as this approach allows for predicting an empty set on null queries, which neither a generator prompted to read sources from the query text nor a substring matcher can do.

Abstract

Multi-Meta-RAG improves retrieval for multi-hop question answering by filtering a vector store on metadata, specifically the news source and the publication date, which it extracts from each query using GPT-3.5. This extractor requires a proprietary API call for every query, and its output is not restricted to the allowed sources: even though the prompt specifies 49 sources, GPT-3.5 generates 128 distinct source names, 79 of which do not appear on the list. We show that the source part of this extractor can be replaced by a local, deterministic probe trained on the hidden states of a small open-source language model. The probe is a multi-label logistic classifier on the mean-pooled hidden states of one layer, trained with a class-balanced loss on the gold evidence sources. Using 5-fold cross-validation to select the layer, we evaluate four models ranging from 135M to 1.5B parameters out-of-fold across all 2556 MultiHop-RAG queries. The probe reaches a set-exact accuracy of 90.9%, compared to 88.0% for a substring baseline and 80.9% for GPT-3.5. The difference comes from null queries, for which GPT-3.5 never predicts an empty source set. Across the remaining queries, all three methods achieve set-exact accuracies within 1.1 percentage points of each other. Unlike earlier probing work, we use the probe as a component of a retrieval pipeline rather than for interpretation. For all four models, the cross-validation selects one of the first layers, in contrast to the middle layers reported for retrieval tasks. We attribute this to the source name appearing verbatim in 95.4% of gold query-source pairs. The 135M model is 1.5 percentage points less accurate than the 1.5B model, so the filter requires only a partial forward pass through a small model and one linear layer. The code is available at https://github.com/mxpoliakov/Multi-Meta-RAG.