Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of- K Selection

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Goal Retrieval versus Action-Value Ordering".

Jane: Here's the gist of the paper. The gist The contrastive critics remain useful compatibility rankers on navigation and manipulation tasks, but action selection requires a value-calibrated scalar<ref:2607.27422#pg12> How it works Contrastive critics are widely used in goal-conditioned reinforcement learning (Eysenbach et al., 2022) They embed state–action pairs and goals and produce a value-like score for retrieval, action ranking, planning, and candidate selection The standard bilinear critic is fθ(s, a, g) = ϕθ(s, a)⊤ψθ(g) Contrastive training pulls state–action embeddings toward reachable goals and away from negatives Failure Mechanisms The raw bilinear critic exposes one mechanism through the decomposition fθ(s, a, g) =∥ϕθ(s, a)∥∥ψθ(g) cosϕθ(s, a) Two mechanisms can increase the score: better angular alignment and state–action embedding norm growth Off-support nonidentifiability allows the critic to be well determined on the training support and arbitrary elsewhere This freedom allows a bilinear critic to have unbounded behavior outside the support Candidate Maximization and Regret Candidate maximization connects these facts to regret by comparing it with a pool oracle The candidate maximization selects a⋆K = arg max i≤K fθ(s, ai, g) The pool oracle selects the highest-value action among the same K draws, denoted as a⋆Q = arg maxi≤K Q⋆(s, ai, g) The resulting candidate regret is RegretK = Q⋆(s, a⋆Q, g) − Q⋆(s, a⋆K, g) ≥ 0 Proposition 2 shows that if an off-support bad set dominates the score tail with probability p ∈ (0, 1), then lim inf K→∞ E

RegretK: ≥ qgood − qbad Score Semantics and Calibration The score can increase through closer alignment, a larger state–action embedding norm, or false alignment outside the data support In-support compatibility ranking does not guarantee value ranking (Section S1) The cosine and hybrid readouts have near-zero or negative rank correlation with value (Table 2) TD-Q produces strong progress ordering (τ = 0.49–0.79), whereas raw bilinear has only weak, task-dependent ordering Experimental Findings The experiments distinguish between two hypotheses: H1 (norm inflation) and H2 (value decalibration) The results support H2, showing that the top score tail remains poorly calibrated to value even when the score is bounded The controlled 2D environment verifies both candidate-level stages: high-score false positives exist, and stronger search finds them more often TD-Q remains near the oracle in the controlled toy, while raw bilinear selects off support on at least 80% of queries The training/readout control locates the failure in the objective, showing that raw-trained embeddings retain weak ordering after normalization, whereas cosine training removes it The contrastive critics thus keep goal-compatibility information but lack reliable graded-progress ordering, especially in the score tail targeted by candidate search The next section measures how this gap affects selection Contrastive critics remain useful rankers, while action selection benefits from value ordering over searched actions How it works The training/readout control locates the failure in the objective, showing that raw-trained embeddings retain weak ordering after normalization, whereas cosine training removes it The separate variant gives qTD its own encoder.

Tom: Both heads in the joint variant share the contrastive backbone, which also receives the value gradient This control shows that selection failure lies in the deployed score Score Semantics and Calibration The readout comparison shows two roles of magnitude: the norm∥ϕ∥ carries little value signal within support, but raw bilinear magnitude can still provide an off-support extrapolation channel under candidate maximization The cosine and hybrid readouts select low-value OGBench actions with negative cosine values (median −0.25 to −0.74), and no selected score exceeds 0.9 Selection Consequences of Score Tail Search The selection exposes this gap because it optimizes the critic over the full candidate set instead of evaluating only training data TD-Q remains near the oracle, while contrastive critics have roughly an order of magnitude more regret at K=64 The sweep separates search pressure from miscalibration: selection worsens only when the score tail lacks calibration Training and Deployment Scope The training/readout control locates the failure in the objective, showing that raw-trained embeddings retain weak ordering after normalization, whereas cosine training removes it We therefore argue against maximizing an uncalibrated compatibility score, not contrastive representations or expressive policies Future work can test when controller self-correction makes score misordering inconsequential Score Semantics and Calibration Compatibility ranking need not imply value ranking (Section S1) The contrastive critics perform well on the task they are trained for: retrieving the goal TD-Q generally shows the reverse pattern: stronger γd ordering and less emphasis on retrieval Compatibility and value ranking can therefore diverge within the same critic Retrieval Metrics Raw and cosine reach retrieval AUCs of 0.96–1.0, while hybrid is lower and more variable at 0.86–0.99 The contrastive readouts remain the strongest retrievers on every task under that construction Final Summary The contrastive critics thus keep goal-compatibility information but lack reliable graded-progress ordering, especially in the score tail targeted by candidate search The next section measures how this gap affects selection Contrastive critics remain useful rankers, while action selection benefits from value ordering over searched actions How it works The two-head critic with calibration and retrieval in one network shows that a single network can provide both calibrated selection and near-perfect retrieval The separate variant matches TD-Q’s γd ordering on all three tasks while keeping cosine-level retrieval This control shows that the selection failure lies in the deployed score Score Semantics and Calibration The pathwise Bellman consistency along actions in the dataset is zero, whereas the contrastive scores are 4–8× larger, ranging from 0.2 to 0.38 TD-Q’s low error is expected from its Bellman training, so the contrastive errors are the relevant comparison The TD-Q row is partly circular because the deployed td q critic and the reference both use GCIQL The contrastive rows provide the relevant comparison Retrieval Metrics Table S10 reports per-task retrieval AUC and recall@1, 5 for each deployed score Contrastive critics retrieve well despite weak γd ordering (Table 2), while TD-Q generally retrieves less well Raw and cosine retrieve nearly perfectly in both domains Score Semantics and Calibration The pathwise Bellman consistency along actions in the dataset is zero, whereas the contrastive scores are 4–8× larger, ranging from 0.2 to 0.

Paper summary: Tom: Let’s get into the specifics of "Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of-K Selection." The paper argues that goal retrieval methods, which use contrastive critics to find actions compatible with a goal, are not inherently safe when you use them for best-of-K selection because they don't provide reliable ordering based on the true value of those actions >

Jane: So the thesis is that these critics can be used as objectives for planning or generating candidates, but they can also get tricked into prioritizing actions based on how well their embeddings align with the goal, rather than their actual expected return from a policy >

Lu: They show two mechanisms that increase the score: better angular alignment and a larger state–action embedding norm growth. This means you can have high scores even if the underlying value is poor or if you're far outside the data support where nonidentifiability allows for unbounded behavior >

Meng: So, even if we get a high score, that doesn't mean the action is good for production because it could be an artifact of how the critic was trained on specific data points >

Lalam: The paper compares these to other methods. They look at raw and cosine critics, and they find that while raw critics reach retrieval AUCs of zero point nine six to one point zero zero, their progress ordering on navigation tasks is much weaker and sometimes even inverted > <ref:2607.27422#pg1>

Tom: And the most telling result for me is in the exact-value controlled problem where all three contrastive selectors have higher mean regret than just picking randomly >

Jane: That regret can even increase when you increase K, which shows that as you search more candidates, the lack of value ordering gets worse >

Lu: The paper confirms that TD-Q, which is trained with value supervision, stays close to the pool oracle in this controlled toy problem >

Tom: And pointMaze was the only task where they found a reliable learned value reference for TD-Q, showing that in some environments you can get lucky with the training setup >

Jane: So what does this imply? It suggests that while these critics are great at finding compatible actions, you need something else—a graded value supervision—to guide the final selection process properly >

Conclusion: Tom: Looking at "Goal Retrieval versus Action-Value Ordering: Contrastive Critics under Best-of-K Selection," the authors are essentially saying that using these contrastive critics as a direct objective for best-of-K selection is risky because the score isn't calibrated to value >

Jane: So, in simple terms, it means you can have an action that looks perfect according to your compatibility metric but ends up being terrible in practice if you don't check its actual value first >

Lu: The title itself points out the tension: goal retrieval versus action-value ordering. It’s about whether getting a good goal matches actually having a good result from the policy >

Meng: So, for deploying models, this means we shouldn't just trust the highest compatibility score when picking candidates; we need an extra step to order them by their value before deployment >

Lalam: I think what they really want us to take away is that contrastive critics remain useful as rankers for finding goal-compatible actions, but action selection needs a value-calibrated scalar instead >

Tom: Yeah, so the next time we see a paper using these contrastive methods for selection, we need to be careful. High retrieval performance doesn't prove that the critic ranks actions correctly for best-of-K selection >

Jane: It means if you’re building a system around this, you need a separate action-ordering evaluation before you put it into production >