Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

arXiv:2610.12830 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Cross-Behavior Policy Optimization".

Jane: As an excellent, fastidious, and diligent AI researcher, I have meticulously analyzed the provided text excerpt from arXiv.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper right now called Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target. It’s about how we use all our existing data, both the offline demonstrations and the rollouts we generate ourselves, in one go.

Jane: That sounds like it could be a big deal because right now, they seem to be separate things—you do supervised fine-tuning first and then reinforcement learning later—which means you can't really reuse all that collected data easily.

Lu: Exactly. The paper proposes this idea where they make these sources compatible through one regression loss using a fixed reference target. They claim that because the generating policy doesn't enter this target, positive offline completions, negative ones, reference responses, and current and historical rollouts can share a common record format and replay buffer without needing to coordinate separate SFT and RL objectives.

Meng: I’m thinking about what this means for deployment. If you can just throw everything into one buffer and optimize with the same objective, does that make the training pipeline way simpler? Or does it introduce some weird instability when we're running things at scale?

Lalam: Based on what I see in this paper, CBPO-Unified combines offline demonstrations, on-policy rollouts, and off-policy rollouts into one optimization loop using a shared regression target. This means we don't have to store separate behavior policy probabilities or importance weighting anymore.

Tom: Right, so the core idea is that they create this fixed reference target that doesn't depend on the behavior policy itself. They define it as a combination of the completion’s reference-policy log-probability, scaled verifier reward, and a frozen prompt-level normalizer.

Jane: That sounds like they’re trying to anchor the optimization around something stable while still steering toward higher rewards based on that verifier reward. What is the actual math they are using for this target?

Lu: They define it mathematically as "bt(x, y) equals log pi ref of y given x plus r(x, y) beta minus log Z ref of x". And then they fit the model's log-probability to that target using a response-level loss. That structure is what keeps the objective consistent across all data regimes.

Meng: If it keeps the objective consistent, does that mean we don't have to worry about one data type overpowering another when we mix them? I worry about cold starts and having the wrong kind of feedback during early training phases.

Lalam: The advantage they point out is that this allows optimization to retain the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. It simplifies the coordination aspect significantly.

Paper summary: Tom: They also show variants: CBPO-Offline, CBPO-OnPolicy, CBPO-OffPolicy, and finally CBPO-Unified. This unified version is what they are testing to see if it truly combines everything effectively into one process.

Jane: And looking at the results on Qwen3 model scales and four mathematical reasoning benchmarks, they claim this approach delivers leading average success rates for both avg@thirty-two and pass@thirty-two across all those different data regimes. That’s a pretty broad claim across different setups.

Lu: They show that on Qwen3-1 point 7B, CBPO-Unified improves benchmark-averaged avg@thirty-two by two point three three percentage points and pass@thirty-two by four point four four percentage points over the SFT DAPO baseline, which is a significant lift compared to the previous method.

Meng: Those are solid numbers for a model of that size, but what about scaling up? If you take Qwen3-8B, does this advantage hold up as much when we're dealing with larger models that eat more compute?

Lalam: The paper shows that on Qwen3-8B, the unified approach leads all eight reported benchmark-metric combinations for that scale. Plus, when they looked at historical rollouts specifically for CBPO-Unified, retaining those earlier rollout rounds improved metrics by one point four nine percentage points and two point zero seven percentage points over the no-history configuration.

Tom: So, if we look at the cost side of things, I need to know if this is just a theoretical win or if it's actually practical for production environments where we have to run models on real traffic.

Meng: That’s my main concern. They say on Qwen3-1 point 7B, CBPO-Unified requires one hundred nineteen point three six GPU hours, which they claim is within two percent of DAPO, REINFORCE++, GSPO, and TBRM. And for the Qwen3-8B model, it takes four hundred thirty-seven point four four GPU hours, which they put below non-PPO on-policy baselines and less than half the cost of PPO.

Jane: It’s interesting that they compare it against PPO costs because PPO usually requires a lot of coordination between policy and value functions, which is what CBPO seems to skip by using this shared target. It sounds like the efficiency comes from not having to store or coordinate those extra components.

Lu: The paper also points out the fragmentation in conventional post-training pipelines, where SFT and RLVR are separate objectives that exclude negative demonstrations from SFT and retire positive demonstrations once RLVR begins. CBPO directly addresses that fragmentation by creating a shared regression target for all three data regimes.

Tom: So to sum up the main point of this paper, Cross-Behavior Policy Optimization is a framework that unifies learning from offline demonstrations and on- and off-policy rollouts through one regression loss using a fixed reference target. This makes these sources compatible without needing separate SFT and RL objectives or storing complex policy probabilities.

Jane: And what does that mean for the bigger picture? If we can combine these data streams this smoothly, it suggests a way to leverage much more of the data we’ve already collected before we start training our final policies.

Paper summary: Lalam: It means we can treat positive and negative offline completions, reference responses, and current or historical rollouts as part of one unified record for optimization. This makes the learning process much more holistic across different data types.

Meng: If it works on this scale, it suggests that the way we structure our training data—mixing demonstrations with self-generated data—can be much more efficient than sticking to a strict two-stage pipeline where you have to discard or switch objectives midway.

Tom: The authors are showing that by restricting the available sources to offline, on-policy, and off-policy variants, they can see how this works before moving to the full unified version. It’s a methodical way of proving it step by step.

Jane: They also flag a limitation in their work—the paper states that conventional reasoning post-training commonly follows a two-stage pipeline where SFT is followed by RLVR, and CBPO directly addresses how that fragmentation happens, but they don't necessarily claim to fix every single data usage problem out there.

Lu: That’s right; the limitation they state is about what their method doesn't do, which is that it doesn't necessarily replace every existing post-training setup in the field with this single framework. It’s a specialized unification of existing techniques.

Tom: So, while CBPO-Unified leads all reported benchmark metrics and shows strong efficiency gains compared to SFT DAPO—gaining two point three three percentage points in avg@thirty-two on Qwen3-1 point 7B—the authors are being careful not to overstate its general applicability across every single post-training setup for reasoning.

Jane: So what does this imply for people who only listen to the show? It means that if you’re working with mixed data sources, like having both good offline examples and ongoing rollouts, you might have a more direct path to better performance by unifying your objective instead of keeping them in separate loops.

Lu: The implication is that the structure of the loss function—the fixed reference target—is what makes this work across these different data regimes together. It’s about finding a single mathematical way to guide the policy toward high rewards regardless of whether the data came from an offline demonstration or a current rollout.

Meng: For someone shipping models, it means less plumbing to manage when you want to combine older data with new generation cycles, because the replay buffer handles it without needing complex coordination between separate SFT and RL objectives.

Lalam: I’m confident in what the paper shows about their unified model, as it demonstrates strong results across different model scales and benchmarks. The result is that CBPO-Unified is empirically shown to be the most effective configuration presented in this study.

Conclusion: Tom: So, Cross-Behavior Policy Optimization is this new framework that tries to make offline data and online rollouts work together using one single loss function based on a fixed reference target.

Jane: That’s right, and it uses a shared regression target, which is the mathematical anchor that lets all those different types of data share the same objective.

Lu: The authors focus on unifying these sources so you don't have to coordinate separate supervised fine-tuning and reinforcement learning objectives anymore.

Meng: So what does this mean for deployment? Does it just make the training pipeline smoother or is there some hidden complexity when we try to run this on real traffic?

Lalam: It means you get a common reward-labeled record format for everything, which simplifies the replay buffer and removes the need to store behavior policy probabilities or importance weighting.

Tom: I’m looking at the title of this paper, Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target. It sounds like they are tackling that fragmentation head-on.

Jane: Yeah, it's about taking those two separate learning paths—offline demonstrations and policy rollouts—and finding a single way to train the model using one regression loss.

Lu: The authors show this unification across offline, on-policy, and off-policy variants before they show the full unified version. It’s a methodical approach.

Meng: And looking at the results they show, it looks like CBPO-Unified is the most effective configuration because it combines all those data sources.

Tom: On Qwen3-one point 7B, this unified version improved benchmark-averaged avg@thirty-two by two point three three percentage points and pass@thirty-two by four point four four percentage points over the SFT DAPO baseline. That's a real lift for a smaller model, so that’s interesting to me as an engineer.

Jane: It really shows how much better performance can be when you are able to pull in both old data and new generation cycles without having to switch objectives midway through the training process.

Lu: They also show that retaining historical rollouts specifically improves metrics by one point four nine percentage points on Qwen3-8B for the unified version, which is a good result when you're talking about model scales.

Meng: The cost data they provided is pretty telling too; on Qwen3-8B, it took four hundred thirty-seven point four four GPU hours for the unified approach, which they positioned as being less than half the cost of PPO. That’s something I can actually think about in terms of what we can afford to run at scale.

Tom: So if you only listen to this show, remember this: CBPO is a framework that unifies learning from offline demonstrations and on- and off-policy rollouts through one regression loss using a fixed reference target.

Jane: It’s about creating a single mathematical way to guide the policy toward high rewards regardless of whether the data came from an offline demonstration or a current rollout.

Lu: The implication is that you can treat positive and negative offline completions, reference responses, and historical rollouts as part of one unified record for optimization.

Meng: It means less plumbing to manage when you want to combine older data with new generation cycles because the replay buffer handles it without needing complex coordination between separate SFT and RL objectives.

Tom: This suggests that by structuring your training data this way, you can be much more efficient than sticking to a strict two-stage pipeline where you have to discard or switch objectives midway.

Jane: The authors are being careful not to claim this replaces every existing post-training setup in the field with this single framework; it's a specialized unification of existing techniques.

Lu: They also flag that the method doesn't necessarily fix every single data usage problem out there, which is a fair limitation to acknowledge.

Tom: So while CBPO-Unified leads all reported benchmark metrics and shows strong efficiency gains compared to SFT DAPO—gaining two point three three percentage points in avg@thirty-two on Qwen3-one point 7B—the authors are being careful not to overstate its general applicability across every single post-training setup for reasoning.

Jane: So what this implies for people working with mixed data sources is that you might have a more direct path to better performance by unifying your objective instead of keeping them in separate loops.

Lu: The structure of the loss function—the fixed reference target—is what makes this work across these different data regimes together. It’s about finding a single mathematical way to guide the policy toward high rewards regardless of the data source.

Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: As an excellent, fastidious, and diligent AI researcher, I have meticulously analyzed the provided text excerpt from arXiv.

Key concepts

Fixed-Reference Target: This is a mathematical target derived from an optimal policy that does not depend on the specific behavior policy generating the data. It combines the log-probability of a reference policy, a scaled reward, and a frozen prompt normalizer to create a consistent objective for all training sources.
Shared Regression Loss: CBPO uses one regression loss function to train the model. This loss is applied consistently across offline demonstrations, current rollouts, and historical data. Because the target is fixed relative to the reference policy, this single objective guides the policy toward high rewards regardless of where it was sampled from.
Cross-Behavior Policy Optimization (CBPO): CBPO is a framework that makes offline demonstrations and various rollouts compatible by using a common reward-labeled record format. It avoids coordinating separate SFT and RL objectives or storing complex probabilities, instead using one unified loss to optimize the policy across all data types.

Summary

As an excellent, fastidious, and diligent AI researcher, I have meticulously analyzed the provided text excerpt from arXiv. My task is to synthesize this information into a comprehensive, long, and detailed summary of Cross-Behavior Policy Optimization (CBPO).

Here is the detailed research summary:

Detailed Research Summary: Cross-Behavior Policy Optimization (CBPO)

Cross-Behavior Policy Optimization (CBPO) is a novel framework designed to unify the training objectives for offline demonstrations and on/off-policy rollouts by employing a single regression loss driven by a fixed reference target. This unification allows disparate data sources—offline demonstrations, on-policy rollouts, and off-policy rollouts—to share a common reward-labeled record format and replay buffer without requiring complex coordination between separate Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) objectives, nor the storage of behavior policy probabilities or importance weighting.

Core Mechanism: The Fixed-Reference Regression Target

The fundamental innovation of CBPO lies in its construction of a fixed-reference regression target. This target is derived from a closed-form optimal policy under KL regularization and is crucially independent of the behavior policy (mu) that generated the completion. This independence is achieved by constructing a shared regression objective that combines three key components:

Completion’s reference-policy log-probability: pi ref(y x)
Scaled verifier reward: r(x, y) beta
Frozen prompt-level normalizer: Z ref(x)

The target is mathematically defined as:

bt(x, y) = pi ref(y x) + r(x, y) beta − Z ref(x).

CBPO then fits the model's log-probability (pi theta) to this target using a response-level loss:

CBPO(x, y; theta):= pi theta(y x) over pi ref(y x) − r(x, y) beta + Z ref(x)

This structure ensures that the optimization objective remains consistent across all data regimes (offline, on-policy, off-policy), allowing a single regression loss to guide the policy toward higher rewards.

Data Compatibility and Variants

CBPO achieves its unifying power by leveraging a common reward-labeled record format for all inputs. This enables three distinct variants:

CBPO-Offline: Utilizes only offline demonstrations and reference responses.
CBPO-OnPolicy: Trains exclusively on current rollouts generated by the policy being optimized.
CBPO-OffPolicy: Combines reference responses with both current and historical rollouts.
CBPO-Unified: The comprehensive variant that integrates all three sources (offline demonstrations, on-policy rollouts, and off-policy rollouts) into a single optimization process.

Performance and Empirical Results

The efficacy of CBPO is demonstrated across two model scales (Qwen3-1.7B and Qwen3-8B) and four distinct mathematical reasoning benchmarks.

Key Performance Metrics:

Avg@32: Measures the average success rate over 32 samples.
Pass@32: Measures the probability that at least one of 32 sampled completions is correct.

Superiority Over Baselines (SFT to DAPO):

CBPO-Unified consistently outperforms the standard SFT to DAPO baseline, showing a more direct path toward higher reward regions. For instance, on Qwen3-1.7B, CBPO-Unified improves benchmark-averaged Avg@32 by 2.33 percentage points and Pass@32 by 4.44 percentage points.

Comparative Benchmarking:

Against BRIDGE: CBPO-Unified exceeds BRIDGE performance by 1.47/3.90 and 2.34/2.56 points, respectively, across the two model scales (Avg@32 and Pass@32).
Against Other Methods: The unified approach leads all eight reported benchmark-metric combinations for Qwen3-8B.

Historical Replay Benefit:

Retaining historical rollout rounds further enhances performance. For CBPO-Unified, retaining earlier rollout rounds improves aggregate metrics by 1.49/2.07 points, leading to a final overall improvement over SFT to DAPO of 2.33/4.44 points.

Cost Efficiency:

CBPO-Unified demonstrates significant computational efficiency:

On Qwen3-1.7B, it requires 119.36 GPU hours, which is within 2% of DAPO, REINFORCE++, GSPO, and TBRM.
On Qwen3-8B, it requires 437.44 GPU hours, positioning it below non-PPO on-policy baselines and less than half the cost of PPO.

Conclusion

Cross-Behavior Policy Optimization (CBPO) successfully bridges the gap between offline data utilization and online policy optimization by establishing a shared regression target that is invariant to the behavior policy. By unifying offline demonstrations, on-policy rollouts, and off-policy rollouts under one regression loss, CBPO delivers state-of-the-art performance across diverse data regimes. The unified variant (CBPO-Unified) is empirically shown to be the most effective configuration, leading all reported benchmark metrics and offering substantial gains over traditional SFT to RLVR methods by following a more direct trajectory toward high-reward policies.

Improvements for AI systems

Cross-Behavior Policy Optimization (CBPO) enables compatible data reuse by applying one regression loss to Offline Demonstrations and On- and Off-Policy Rollouts through common reward-labeled records, which allows optimization to retain the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic.
CBPO enables significant performance gains over conventional methods; for instance, on Qwen3-1.7B, CBPO-Unified improves benchmark-averaged avg@32 and pass@32 over SFT→DAPO by 2.33 and 4.44 percentage points, respectively.
The method supports data reuse across regimes via a shared target, as the paper states that Neither expression contains µ, meaning two records with identical logged fields induce the same per-sample loss and gradient even if they were collected by different behavior policies.
CBPO-Unified demonstrates superior optimization trajectory geometry compared to SFT→DAPO, showing that it follows a more direct projected path toward a higher-reward region (L/D = 1.0205, 58.20◦) rather than the sharp turn observed in the conventional pipeline.
The system can utilize historical rollouts effectively by employing replay strategies such as Full replay or CBPO-Unified FIFO-1, which, when used with appropriate age and residual level selection, can improve performance by over 2.33/4.44 points compared to the SFT→DAPO baseline.
The frozen prompt normalizer's bias is bounded at finite sample sizes by 0 ≤ log Zref(x) − E[log Zbref,N(x)] ≤ σ 2 x 2N a 2 x, where the bias is O(1/N) and converges to zero as N → ∞.
The system can be tuned for specific data regimes by restricting the trainable sources; for example, CBPO-Offline fixes Doff ∪ Dref and performs no rollout collection, which allows for regime-matched performance improvements over baselines.