Bayesian Preference Learning for Test-Time Steerable Reward Models

arXiv:2602.08819 · cs.LG, cs.CL · Submitted 2026-02-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Bayesian Preference Learning for Test-Time Steerable Reward Models".

Jane: The gist: Variational In-Context Reward Modeling (ICRM) is a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called Bayesian Preference Learning for Test-Time Steerable Reward Models. The authors are Jiwoo Hong, Shao Tang, and Zhipeng Wang from LinkedIn Corporation and Nubank.

Jane: It sounds like they’re tackling a problem where reward models, which we use to align language models with human preferences through reinforcement learning, get stuck once they’re trained. They become static at test time.

Lu: Exactly. The paper proposes this Variational In-Context Reward Modeling objective called ICRM to fix that. It casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a Beta prior #pg1

Meng: So what's the main claim here? What exactly is ICRM doing that’s different from just training a standard reward model?

Tom: Basically, ICRM uses in-context preference demonstrations to make the reward model steerable at test time. It lets it adapt to unseen preferences. They show it works for both single and multi-objective settings #pg2

Jane: So, how does this work mechanically? The summary says they approximate the true preference distribution with a Beta posterior conditioned on those in-context demonstrations.

Lu: They use a Beta prior on the Bradley-Terry model, which is a conjugate prior #pg3. This yields a closed-form training loss via variational inference #pg2

Tom: It’s about mapping inputs to the parameters of this approximate posterior distribution, specifically using alpha q = mu tau and beta q = (one - mu) tau, where mu is the posterior predictive probability and tau controls concentration #pg2 <ref:2602.08819#pg1>

Meng: And what's the goal of that KL regularization term they mention? Is it just for stability?

Lu: It’s to prevent excessive maximization of the preference mean on training data, which ensures we get an interior optimum #pg3. Theorem three point two proves that every global minimizer mu satisfies zero < mu < one and zero < tau < infinity #pg3

Jane: That sounds like a lot of math to keep track of, but what does this mean for the person using the AI model? What’s the practical difference?

Tom: The practical difference is test-time steerability. If you have preferences you haven't seen before, ICRM allows the reward model to shift toward those new preferences using just a few examples in context #pg2

Meng: I'm interested in what they say about performance. They mention improving RM-Bench accuracy from sixty point five to seventy point eight with more demonstrations #pg1 <ref:2602.08819#pg1,RM-Bench accuracy from 60.5 to 70.8>

Lu: That improvement shows it adapts to unseen preference distributions at test time for both single and multi-objective settings #pg1

Jane: And they also mention achieving lower calibration error than a generative judge on moral dilemma preferences #pg1 That’s important because those are tricky scenarios.

Tom: They also show that with more demonstrations, ICRM improves RM-Bench accuracy from sixty point five to seventy point eight and expands the attainable Pareto frontier under conflicting preferences #pg1 <ref:2602.08819#pg1,with more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to>

Meng: Okay, so on the practical side, they also applied it to verifiable rewards in reinforcement learning. How did that go?

Lu: In RL with verifiable rewards, ICRM outperforms both the verifier and a conventional reward model in math reasoning #pg1 They reach fifty-three point eight percent MATH-five hundred accuracy #pg1

Jane: Fifty-three point eight percent on a math benchmark compared to fifty percent and fifty-one point four for the other methods. That’s a solid gain if it works reliably on real tasks.

Tom: And they showed that this alignment with accuracy is stronger when using lambda = zero point one in RLVR training #pg1 The correlation analysis indicates that ICRM with lambda = zero point one has a stronger alignment with the accuracy #pg1

Meng: So, what are the limitations? What doesn't this framework handle well?

Jane: They do flag that the trained ICRM could be steered to prefer harmful behaviors when tested against flipped labeled safety preference datasets #F

Lu: That’s a fair point. The paper notes that while it offers wider applications for RLHF, like steering to harmful behaviors, those things need appropriate safeguards and restricted access to unsafe steering demonstrations #F

Tom: So we have this objective that allows us to adapt the model later in the process using context, and it has theoretical guarantees against over-optimization through that KL regularization term #pg3

Jane: It seems like they’ve built a principled way to make reward models flexible instead of just static after training. That's what this paper is about.

Lu: The core contribution is providing a framework grounded in Bayesian views for preference learning that unifies in-context steerability with regularization #G

Tom: So, to wrap up on the paper "Bayesian Preference Learning for Test-Time Steerable Reward Models," it introduces ICRM. It’s this new objective that lets reward models adapt to preferences they haven't seen during training by using in-context preference demonstrations #pg1

Jane: The authors are showing that this approach works for both single and multi-objective settings, improving accuracy on benchmarks like RM-Bench from sixty point five to seventy point eight with more demonstrations #pg1 <ref:2602.08819#pg1,for both single and multi-objective settings>

Lu: The paper proves that this variational design admits a global interior optimum with finite confidence, which is guaranteed by the KL regularization term #pg3

Meng: It's about making models more adaptable post-training without needing full retraining or fine-tuning for every new preference standard.

Tom: Exactly. It gives us a way to encode complex, graded preference mixtures at test time and offers theoretical control over how much the model drifts based on the context it sees #pg1

Conclusion: Tom: So we've been talking about ICRM, this Variational In-Context Reward Modeling objective that lets reward models adapt to preferences they haven't seen during training #pg1

Jane: Exactly, Tom. It’s about making those static reward models flexible at test time using preference demonstrations #pg2

Tom: The authors are Jiwoo Hong, Shao Tang, and Zhipeng Wang from LinkedIn Corporation and Nubank #pg1 They put this Bayesian approach to reward modeling into action #pg3

Jane: So what does that title really mean for someone just listening? It means we’re looking at a way to stop reward models from being locked in their initial training data once they’re deployed #pg2

Tom: It suggests that instead of retraining the whole model every time we want it to prefer something slightly different, we can use context—like a few examples—to steer its behavior dynamically #pg1

Jane: That's the core idea. It’s not about retraining, it’s about having a mechanism for real-time preference adjustment based on the immediate context of the task #pg2

Tom: They show this works for both single and multi-objective settings, which is key because real preferences are rarely just one thing #pg1

Jane: And they have theoretical guarantees that this variational objective actually has a stable, interior solution, meaning it won't just wildly jump around during test time #pg3

Tom: That stability comes from the KL regularization term they use to control how much the model drifts away from its initial training #pg3

Jane: It gives us a principled way to temper that drift. It’s a step toward reward models that aren't just brittle after training #pg2

Tom: We saw some great results in reinforcement learning, too, where this ICRM approach actually beat the conventional verifier for math reasoning on MATH-five hundred #pg1

Jane: Yeah, so while it’s a fancy Bayesian trick to make models more flexible, the real impact is seeing how much better they can perform when you give them just a little bit of in-context guidance #pg2

LinkedIn Corporation · Nubank

cs.LG, cs.CL

Submitted: 2026-02-09

Updated: 2026-10-19

Comments: NeurIPS 2026

Code: https://github.com/huggingface/trl

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: The gist: Variational In-Context Reward Modeling (ICRM) is a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations.

Key concepts

Variational In-Context Reward Modeling (ICRM): ICRM is a novel Bayesian objective that treats reward modeling as amortized variational inference. It allows a reward model to infer the true preference distribution at test time by conditioning on a few in-context preference demonstrations, making the model steerable without full retraining.
Bradley-Terry Model: This is a standard statistical model used to represent pairwise preferences between items. In this context, it assumes that the probability of preferring one item over another is determined by their scores, allowing ICRM to learn and approximate complex preference distributions.
Test-Time Steerability: This refers to the ability of a reward model to dynamically change its behavior or preferences when presented with new context during inference. ICRM achieves this by using in-context demonstrations as evidence to adjust the inferred preference distribution on the fly.

Terminology

Verifiable Rewards — 2× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode
Adaptability — 1× in this paper · explained in 1 episode
Bradley-Terry model — 1× in this paper · explained in 3 episodes
Calibration — 1× in this paper · explained in 9 episodes
KL Regularization — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
Theoretical Guarantees — 1× in this paper · explained in 2 episodes
Variational In-Context Reward Modeling — 1× in this paper

Summary

The gist: Variational In-Context Reward Modeling (ICRM) is a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations.

How it works

The ICRM framework casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior The central idea is to approximate the true preference distribution with a Beta posterior conditioned on in-context preference demonstrations This allows ICRM to learn preferences in-context with few-shot demonstrations, enabling test-time steerability of a classifier RM that can dynamically adapt to one or more mixtures of arbitrary preferences.

The model parameterization involves mapping the inputs to the parameters of an approximate posterior distribution, specifically a Beta distribution parameterized by αq = µτ and βq = (1 − µ)τ, where µ is the posterior predictive probability and τ controls concentration. This construction preserves the BT model as a special case because the posterior mean of qθ recovers the BT preference probability, Eqθ[z] = αq/(αq + βq) = µ = σ(uw - ul).

Principled Variational Preference Learning

The method proposes a reward modeling objective that allows RMs to infer test-time preferences from in-context preference samples. A Beta prior is used for the Bradley-Terry model, where the conjugate prior is Beta (α0, β0), and the final learning objective is defined as LICRM(µ, τ; α0, β0). The loss function includes a reconstruction error term measuring how well the approximate posterior explains the observed outcome yw ≻ yl, and a KL regularization term to prevent excessive maximization of the preference mean on training data.

Test-Time Steerability and Optimization

The proposed KL-regularized variational objective directly addresses over-optimization by precluding boundary minima, thus ensuring an interior optimum. Theorem 3.2 proves that every global minimizer (µ⋆, τ⋆) of LICRM(µ, τ; α0, β0) satisfies 0 < µ⋆ < 1 and 0 < τ⋆ < ∞, providing a theoretically guaranteed prevention of reward model over-optimization via preference mean tempering. The dynamic hyperparameter λ(N) controls the KL divergence from the prior, allowing the posterior to transition from high uncertainty when context is minimal (N=1) to a more confident, data-driven posterior as more examples are added.

Empirical Performance and Applications

ICRM demonstrates both calibrated and directional steerability across various settings. In single-objective calibration, ICRM improves RM-Bench accuracy by over 10% point with increasing demonstrations, showing it can encode graded preference mixtures at test time. For multi-objective steerability, the method shows that the posterior mean µ reflects the mixed preference distribution induced by in-context demonstrations rather than collapsing to a single objective. Furthermore, in reinforcement learning with verifiable rewards (RLVR), ICRM outperforms both the verifier and conventional RM in math reasoning, achieving over 2.5% point accuracy on MATH-500.

Key Contributions

The main contributions of the paper are summarized as follows:

Principled variational preference learning: Proposing a reward modeling objective that enables RMs to infer test-time preferences from in-context preference samples and proving that regularizing the Beta posterior with a uniform Beta prior guarantees a global interior optimum.
Single and multi-objective test-time steerability: Showing that the variational design yields both directional and calibrated steerability, extending to multi-objective preference trade-offs.
Parameterizing verifiable rewards with ICRM: Applying ICRM to RLVR where it outperforms both the verifier and the Bradley-Terry reward model on math reasoning, achieving over 2.5% point accuracy compared to the gold verifier.

The paper concludes that ICRM provides a theoretically grounded and practically effective route toward reward models that can adapt to unseen preference standards after training. The final result shows ICRM adapts to both single and multi-objective preference distributions, increasing RM-Bench accuracy from 60.5 to 70.8 with more demonstrations. It also expands the attainable Pareto frontier under conflicting preferences. Finally, in RLVR, ICRM reaches 53.8% MATH-500 accuracy and outperforms both verifier and BT reward model as rewards. The framework offers a principled mechanism for tempering reward over-optimization through the KL regularization term. The paper also provides theoretical guarantees that the variational objective admits a global interior optimum with finite confidence and analyzes how KL regularization mitigates reward over-optimization.

B In-Context Preference Learning Template

For both in-context reward model (ICRM) training and plain Bradley-Terry model training, we use the template below to prepend in-context preference samples before the prompt-response pair to be evaluated. The format includes a [Prompt], [Chosen], and [Rejected] example.

C Training Configurations for ICRM

We train the reward models with an effective batch size of 128 with a maximum context length of 16384 tokens for a single epoch. The training utilizes Flash-Attention 2, FSDP, and Paged AdamW optimizer with 8-bit precision for efficient multi-GPU training.

D Calibrated Steerability Analysis

ICRM trained with λ = 0.1 on Qwen3-4B demonstrates the least ECE and RMSE across different context sizes compared to other configurations. This selection of Qwen3-4B ICRM with λ = 0.1 is chosen for the other steerability experiments. The analysis shows that increasing the number of in-context demonstrations N leads to lower ECE and RMSE values, indicating that ICRM can encode graded preference mixtures at test time.

E Reinforcement Learning with Verifiable Rewards

In RLVR training, ICRM reaches 53.8% MATH-500 accuracy and outperforms both verifier and BT reward model as rewards. The policy trained with ICRM achieves an accuracy of up to 53.8%, whereas it was at most 50.4% and 51.4% for verifier and BTRM cases, respectively. The correlation analysis indicates that ICRM with λ = 0.1 has a stronger alignment with the accuracy, implying that verifiable rewards can also be encoded via ICRM.

F Limitations and Broader Impacts

The trained ICRM could be steered to prefer harmful behaviors when tested against the flipped labeled safety preference dataset. While this offers wider applications of RLHF, such usage should be accompanied by appropriate safeguards, including restricting access to unsafe steering demonstrations and enforcing deployment-time policies.

G A Related Works

Recent work shows that in-context learning (ICL) adapts LLMs to new tasks with few-shot examples, similar to explicit fine-tuning. Prior work has proposed various learning objectives for reward modeling, including active preference acquisition and reward transformations for multi-objective aggregation. The paper introduces a variational in-context reward model that treats reward as a latent random variable and infers a conjugate posterior from a few demonstrations, unifying Bayesian regularization with in-context steerability. The work also explores architectures beyond discriminative BT models, such as generative reward models and critique-out-loud methods.

Improvements for AI systems

Bold preference learning via ICRM: The system can adapt to unseen preference distributions at test time for both single and multi-objective settings because ICRM directly encodes the mean and sharpness of the Beta posterior given in-context preference demonstrations.
Robustness against over-optimization: By employing a KL penalty, the model avoids local optima by ensuring an interior optimum, as proven by Theorem 3.2, which states that "Every global minimizer (µ⋆, τ⋆) of LICRM(µ, τ; α0, β0) defined in (9) satisfies 0 < µ⋆ < 1 and 0 < τ⋆ < ∞."
Calibrated multi-objective steering: The system can navigate conflicting preferences by producing smooth trade-off curves where increasing r raises 'Should Respond' accuracy while reducing 'Should Refuse' accuracy, allowing for continuous test-time navigation of the multi-objective preference space.
Verifiable reward parameterization: For reinforcement learning with verifiable rewards (RLVR), the model can encode math correctness as an in-context preference signal, achieving 53.8% MATH-500 accuracy and outperforming both verifier and BT reward model as rewards.
Contextual confidence calibration: The system's uncertainty is calibrated to the number of demonstrations, where As λ = 0.1 demonstrated the widest confidence range as intended in the variational design, leading to lower calibration error (ECE) on moral dilemma preference pairs.

Abstract

Reward models are central to aligning language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, followed by an ablation study on its role for the test-time steerability, and we analyze how KL regularization mitigates reward over-optimization.