Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling

arXiv:2610.08677 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling".

Jane: The gist The Variance Optimal-CEM estimator selects an interpolation coefficient to minimize variance between OffCEM and DR estimators,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper "Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling," and it tackles that problem where action-level importance weighting in Doubly Robust estimation causes too much variance.

Jane: Right, so you know how standard estimators like DR or IPS are unbiased under certain conditions, but they end up having high variance when you look at the action-level weights? This paper seems to be about finding a way to stabilize that.

Lu: Exactly. The authors introduce an unbiased family of estimators that sits between OffCEM and DR, and then they propose this Variance Optimal-CEM, or VOCEM, which picks the best point in that family based on minimizing variance.

Meng: So if I understand this right, instead of just picking one fixed estimator like DR or OffCEM, VOCEM calculates a specific coefficient to put it somewhere in between those two and makes sure the variance is as low as possible.

Tom: That's the core idea. The paper derives this optimal coefficient in closed form, which means you don't have to guess it with some messy search process; you can just plug in the data from your logging sample to get a plug-in estimate for it.

Jane: And what does that actually mean for us? It suggests that we can get an estimator whose variance is no larger than either the OffCEM or the DR estimator on its own, which is a big deal for stability.

Lu: The family of estimators they define is built around a parameter, let's call it lambda, and they show that the mean squared error decomposes into bias and variance. Since the bias stays zero across this whole family under their assumptions, you just minimize the variance part.

Tom: So they found a quadratic relationship between that interpolation parameter and the variance, which allowed them to find this theoretical minimum point in closed form. That's a very clean mathematical result there.

Meng: From an engineering standpoint, deriving it in closed form is helpful because you can estimate that optimal lambda directly from your sample averages of A and B without needing a lot of expensive simulations.

Jane: But the paper doesn't just stop at the theory; they actually test this VOCEM estimator in fifteen synthetic conditions and eight real-world data settings, showing it beats both endpoints.

Tom: It sounds like you're saying that in those tests, VOCEM consistently showed lower mean squared error than both OffCEM and DR across the board, which is pretty strong evidence for its practical utility.

Lu: The results show a uniform advantage in the synthetic data, where VOCEM's mean squared error is below that of OffCEM in all fifteen conditions tested.

Meng: I see that they also looked at different action spaces; the MSE ratio actually falls as the action space gets bigger, dropping from zero point eight three four with two hundred actions down to four thousand actions. That suggests it handles larger sets of possible actions better than some fixed methods.

Jane: That's interesting, because when you look at real data, like the Wiki10-31K dataset they used, the median performance stays very close to the DR endpoint.

Tom: So it means that while VOCEM is better overall in synthetic setups, its practical advantage on real-world data depends a bit on how complex the underlying action space is and where you are in terms of common support.

Lu: They also looked at a scenario with deficient action support, where the reward model was misspecified; VOCEM stayed below OffCEM throughout that experiment, with relative MSE dropping from about zero point seven six down to zero point five two.

Meng: That robustness under misspecification is important because we often have imperfect reward models in production systems, so if an estimator handles that better, it's more reliable for deployment.

Jane: So they found that VOCEM manages to exploit some of the variance reduction you get from action-level correction while still tolerating some of the bias introduced by that correction.

Tom: That makes sense. It's like you get a little bit of bias, but the massive variance reduction outweighs it in terms of overall mean squared error.

Lu: The conclusion is that VOCEM achieves a variance no larger than either OffCEM or DR, and they propose using an empirical estimate of the optimal coefficient to select the best member of that family.

Meng: From a deployment view, if you can plug-in that coefficient efficiently, it seems like you get a more stable off-policy evaluation without needing to run complex simulations for every policy update.

Jane: So we're looking at an estimator that is theoretically sound, empirically better than both standard options across many conditions, and robust to some model misspecification.

Tom: The paper "Variance-Optimal Off-Policy Evaluation with Conjunct Effect Modeling" shows how choosing the right interpolation point can lead to a lower variance estimator for off-policy evaluation.

Lu: It suggests that adapting the weighting scheme from clusters toward individual actions is a good way to improve estimation when action-level importance ratios in DR are unstable.

Meng: If you're running large scale reinforcement learning, this might mean you can deploy more reliably because the evaluation of your policy is less noisy.

Jane: It’s a lot to digest, but the main point is that by optimizing that interpolation parameter, we get an estimator that balances stability and accuracy across different modeling assumptions.

The paper's summary: Tom: So we're looking at how they tackle that variance problem by interpolating between OffCEM and DR, right?

Jane: Exactly. It’s about finding that sweet spot in the family of estimators where the variance is actually minimized, not just sitting at one of the ends like OffCEM or DR.

Tom: The authors derive this optimal coefficient in closed form, which means we don't need to run a lot of complex searches to find it; we can plug in our data averages directly to get a plug-in estimate for that lambda.

Jane: That’s powerful because it gives us a stable method to pick the best estimator without needing super intensive simulations every single time.

Tom: And what they show is that this Variance Optimal-CEM, or VOCEM, has variance no larger than either OffCEM or DR on its own, which is a big win for stability.

Jane: That’s significant because it means we get better reliability when evaluating a new policy without having to deploy it first.

Tom: They tested this across fifteen synthetic conditions and eight real-world settings, and VOCEM consistently showed lower mean squared error than both endpoints in all those cases.

Jane: That suggests a solid empirical advantage, especially when you look at how it handles different action spaces; the ratio actually drops as the number of actions gets larger.

Tom: And they even looked at scenarios where the reward model was misspecified, showing VOCEM remained below OffCEM and even improved its performance there.

Jane: It seems like VOCEM is robust because it manages to exploit some of that variance reduction you get from action-level correction while still tolerating some of the bias introduced by that correction.

Tom: So it’s not perfect, but they show we can trade a little bit of bias for a lot more stability in the final evaluation number.

Jane: And this points to adapting residual weights based on what we see in our logged data as a really smart way to improve estimation when the action-level importance ratios in Doubly Robust are unstable.

Tom: It’s about taking that theoretical framework and making it practical for deployment by giving us a plug-in formula for that optimal parameter.

Jane: So, if you’re running large scale reinforcement learning and your reward model is messy, this paper suggests VOCEM could be a more dependable tool than just sticking to the standard OffCEM or DR estimators alone.

The paper's improvements: Tom: So we’re talking about how this Variance Optimal CEM estimator actually improves things beyond just being better than DR or OffCEM on paper, right?

Jane: Right, so the authors aren't just giving us a new name for an estimator; they’re showing us *how* to plug in that optimal coefficient lambda.

Tom: The main improvement is this closed-form formula for that lambda, which means we can estimate it directly from our logging sample averages of A and B without needing some super heavy simulations to find the best value.

Jane: That’s what makes it practical; it takes a theoretical optimization problem and turns it into something you can actually compute efficiently on your end.

Tom: And they also showed how this method performs even when the reward model is kinda messed up, in those misspecified regime experiments, VOCEM still stays below OffCEM because of that variance reduction.

Jane: That’s interesting because it implies that the action-level correction part of DR is often unstable, but VOCEM can handle that instability better by balancing it with cluster-level weights.

Tom: The authors point out this trade off: you gain some bias from using the action-level component, but you get a significant reduction in variance overall, which leads to a lower total mean squared error.

Jane: It changes how we think about off-policy evaluation because it suggests that instead of picking one fixed weighting scheme, we should dynamically adjust the weighting based on how much variance we need to cut.

Tom: If you’re running a model where you can estimate those A and B averages quickly, this is a method that makes your evaluation more stable without needing to run massive ensembles just for the estimation step.

Jane: And Lalam says it’s quite sure about this direction because the mathematical structure strongly supports finding that population-optimal coefficient in closed form.

Tom: So, the implication for us on the ground is that we can deploy these evaluation metrics with higher confidence knowing we’ve used a method specifically designed to tame that high variance.

Jane: It opens up a path for more reliable off-policy learning when dealing with real-world data where model dynamics are messy.

Tom: The next thing I want to talk about is how this connects to other papers, because you’ll see a lot of work on variance reduction in the literature.

Conclusion: Tom: So we’re wrapping up on "Variance Optimal Off-Policy Evaluation with Conjunct Effect Modeling" by summarizing what we just heard about finding that closed-form optimal coefficient lambda to minimize variance between OffCEM and DR estimators.

Jane: Exactly, it’s a method that lets us pick the best interpolation point in a family of estimators based purely on minimizing variance, rather than picking one endpoint arbitrarily.

Tom: The implication is we get an estimator whose variance is no larger than either endpoint, OffCEM or DR, which means more predictable results when evaluating new policies.

Jane: That predictability is important because it gives us more confidence in the training loop when we’re dealing with complex off-policy scenarios.

Tom: Lu, what do you think about the closed-form estimation of that lambda? Is that a big step for representation learning or just evaluation?

Lu: It’s a neat mathematical result because it shows that even with complex weighting schemes involving clusters and individual actions, we can find an explicit formula to guide our estimation.

Meng: For me, the closed-form part is good, but what breaks at scale is if we can actually compute those sample averages of A and B in real time on a massive distributed system without huge overhead.

Lalam: I’m sure about the math; it’s sound because it minimizes the mean squared error by directly optimizing the variance term, which is exactly what we need for reliable policy evaluation.

Jane: So, what does this mean for someone just listening to you? It means when you hear about off-policy evaluation in reinforcement learning, you can expect a method that actively tries to reduce noise rather than just using a standard formula.

Tom: Right, and the next thing we could look at is how this relates to other variance reduction techniques we discussed earlier on the show.

Jane: We’ve got some interesting papers coming up about how different AI systems handle uncertainty, which ties right into this idea of choosing the right estimator for a given problem.

Lu: I think it sets a good precedent for future work because if we can optimize coefficients in this way, we could apply similar ideas to other estimation problems where variance is the main bottleneck.

Meng: I just hope that when you take this closed-form plug-in estimate and deploy it, the actual performance on production traffic doesn't suddenly drop because of some unforeseen edge case.

Lalam: I’m confident in its direction; the literature supports this approach to variance optimization for off-policy evaluation very strongly.

Tom: Alright, that’s all we have time for today on "Variance Optimal Off-Policy Evaluation with Conjunct Effect Modeling." We hope you found it useful.

Nicolo Felicioni, Michael Benigni, Maurizio Ferrari Dacrema, Paolo Cremonesi

Politecnico di Milano

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 92/100

The gist: The gist The Variance Optimal-CEM estimator selects an interpolation coefficient to minimize variance between OffCEM and DR estimators, resulting in an estimator with variance no larger than either

Key concepts

Off-policy evaluation (OPE): OPE is the process of estimating how good a new policy will be using data collected while the old policy was in use. Standard methods like Inverse Propensity Scoring (IPS) or Doubly Robust estimation are used to achieve this, but they can suffer from high variance when dealing with action-level importance weights.
Doubly Robust (DR) Estimation: DR estimation is an OPE technique that combines two pieces of information: a reward regression model and importance weights. It uses the regression model as a control variate, meaning it reduces the variance introduced by the importance sampling weights, making it generally unbiased under certain assumptions.
OffCEM: OffCEM is an alternative OPE prior estimator that replaces unstable action-level importance weighting with more stable cluster-level weights. Instead of weighting every single action individually, OffCEM aggregates policy mass into clusters to create a more robust importance ratio.
Variance Optimization (VOCEM): VOCEM finds the best balance between OffCEM and DR by choosing an interpolation coefficient that minimizes the estimator's variance. This optimization leads to a final estimator with variance no larger than either OffCEM or DR, providing superior empirical robustness.

Summary

The gist The Variance Optimal-CEM estimator selects an interpolation coefficient to minimize variance between OffCEM and DR estimators, resulting in an estimator with variance no larger than either endpoint

Off-Policy Evaluation Estimators

Off-policy evaluation (OPE) is the problem of using logged data to estimate the value of a target policy without deployment The standard estimators include Inverse Propensity Scoring (IPS), which multiplies every observed reward by an action-level importance ratio, and Doubly Robust (DR) estimation, which uses a reward regression as a control variate and importance weights only its residual OffCEM replaces action-level importance weighting with more stable cluster-level weights, aggregating policy mass within a cluster to make the importance ratio more stable

The Family of Estimators

The paper shows that DR and OffCEM are endpoints of a common family, where a single scalar parameter interpolates between their residual weights For any fixed estimator in this family, unbiasedness is maintained under the assumptions required by DR and OffCEM The interpolated weight is defined as ωλ(x, a):= wc(x, ϕ(x, a)) + λd(x, a) = (1 − λ)wc(x, ϕ(x, a)) + λwa(x, a)

Variance Optimization

The mean-squared error (MSE) decomposes into squared bias and variance; since the bias is zero throughout the family under Assumption 2, minimizing MSE reduces to minimizing variance The theorem derives a population-optimal coefficient λ⋆ in closed form that minimizes Var(Vbλ(π)) over λ ∈ [0, 1] This resulting estimator, VARIANCE OPTIMAL-CEM (VOCEM), has variance no larger than either OffCEM or DR

Empirical Estimation and Performance

The optimal coefficient λ⋆ can be estimated from the logging sample by computing sample averages for A and B to obtain a plug-in estimate λb Experiments show that VOCEM improves upon both endpoints in all 23 evaluated conditions, exhibiting greater stability and empirical robustness In synthetic settings, the MSE ratio ranges from 0.634 to 0.749, a reduction of 25.1–36.6%

Robustness to Noise and Support Issues

VOCEM demonstrates robustness under various conditions; for instance, in the misspecified reward model regime with deficient action support, VOCEM remains below OffCEM throughout the experiment, with relative MSE decreasing from about 0.76 to 0.52 This robustness is a bias–variance effect where it exploits variance reduction associated with action-level correction while tolerating some of its induced bias

Conclusion

VOCEM addresses excessive variance by adapting the resolution of importance weighting from clusters toward individual actions, while preserving the common unbiasedness property Across 15 synthetic and eight real-world benchmarks, VOCEM achieved lower MSE than both DR and OffCEM, with gains observed under both oracle and learned reward models In the misspecification regime, VOCEM is robust to deficient action support because its low variance allows it to tolerate a moderate amount of support-induced bias while still achieving a net reduction in MSE

How it works

The paper introduces an unbiased family of estimators interpolating between OffCEM and DR using a single scalar parameter λ ∈ [0, 1]
The interpolated weight is defined as ωλ(x, a):= wc(x, ϕ(x, a)) + λd(x, a) = (1 − λ)wc(x, ϕ(x, a)) + λwa(x, a)
The variance is shown to be a quadratic function of the interpolation parameter and its theoretical minimizer λ⋆ is derived in closed form
VOCEM estimates this coefficient directly from the logged data by computing sample averages for A and B to obtain a plug-in estimate λb

Key Findings

**/ The variance is no larger than either endpoint, OffCEM or DR 5 EXPERIMENTS The results on synthetic data show a uniform advantage for VOCEM: its MSE is below that of OffCEM in all 15 conditions, and every pointwise upper confidence bound is smaller than one 5.1.2 RESULTS For the sample size sweep, the MSE ratio increases from 0.188 at n = 500 to 0.873 at n = 8,000, approaching the OffCEM endpoint The benefit becomes larger with the action space: the ratio falls from 0.834 at 200 actions to 4,000 actions 5.2.1 DATASETS AND PROTOCOL Following the real-data evaluation of Saito et al. (2023), we use the EUR-Lex 4K and Wiki10-31K datasets from the Extreme Classification Repository (Bhatia et al., 2016) The action-specific offset ηa controls the separation between these two probabilities: larger values make relevance less easily distinguishable from irrelevance 5.2.2 RESULTS On Wiki10-31K, the median remains essentially at the DR endpoint 5.2.3 RESULTS VOCEM remains below OffCEM in all eight conditions, with every upper confidence bound below one 5.3 D DETAILED EXPERIMENTAL SETTINGS AND ADDITIONAL RESULTS The common-support assumption holds by construction, but Table 3 shows that practical overlap is weak 5.4 E SYNTHETIC EXPERIMENT UNDER DEFICIENT SUPPORT Under the exact reward model, all estimators are approximately unbiased across support levels, so the differences in MSE are driven almost entirely by variance VOCEM exhibits the same qualitative benefit while remaining substantially more stable: its relative MSE decreases from approximately 0.66 at ρ = 0 to 0.13 at ρ = 0.9 The misspecified reward model reveals the complementary bias effect, where VOCEM inherits this bias only through the action-level component 5.5 CONCLUSION Across 15 synthetic and eight real-world benchmarks, VOCEM achieved lower MSE than both DR and OffCEM, with gains observed under both oracle and learned reward models The results support adapting the residual weights to the observed data as a way to improve estimation when DR’s action-level correction is unstable 5.5 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION We introduced VOCEM to improve off-policy evaluation when action-level importance ratios induce high variance in DR estimation The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR 9 CONCLUSION The resulting population estimator has variance no larger than either OffCEM or DR <ref:2610.

Improvements for AI systems

Bold header: Variance-Optimal Estimator Selection in Large Action Spaces

VOCEM can select a residual correction weight, interpolating between action-level (DR) and cluster-level (OffCEM) weighting, to minimize variance by deriving the population-optimal coefficient in closed form. This allows the AI system to achieve greater stability and empirical robustness compared to fixed endpoints.

Bold header: Robustness Across Model Fidelity

The VOCEM estimator demonstrates robustness even when the reward model is imperfect, as shown in Figure 7, where it maintains lower MSE ratios than DR and IPS under learned reward models. This suggests the system can perform reliable off-policy evaluation when facing misspecified reward model scenarios.

Bold header: Handling Deficient Action Support

VOCEM remains robust under deficient action support by balancing bias and variance, as it can exploit the variance reduction associated with action-level correction while tolerating some of its induced bias. This enables reliable evaluation even when the logging policy does not cover all target actions.

Abstract

Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under common support but retains these high-variance action-level weights. A prior estimator, Off-policy evaluation with Conjunct Effect Model (OffCEM), replaces them with more stable cluster-level weights, at the cost of relying on local correctness of the reward model. In this paper, we show that, under the assumptions required by DR and OffCEM, there exists an unbiased family of estimators that interpolates between OffCEM and DR. Building on this result, we propose the Variance Optimal-CEM (VOCEM) estimator, which selects the interpolation coefficient to minimize variance. We derive the population-optimal coefficient in closed form and show that the resulting estimator has variance no larger than either endpoint, OffCEM or DR. Experiments in controlled synthetic settings and on two large-action benchmarks show that VOCEM improves upon both endpoints in all 23 evaluated conditions, exhibiting greater stability and empirical robustness.