CB-Orchestrator: Adaptive Workflow Optimization for LLM Agents via Contextual Bandits

arXiv:2610.13064 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CB-Orchestrator: Adaptive Workflow Optimization for LLM Agents via Contextual Bandits".

Jane: The gist Large Language Model (LLM) agents have demonstrated remarkable capabilities in tackling complex tasks through agentic workflows,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at CB-Orchestrator today. The authors are Jiahang Sun, Zhiwei Shang, Zhipiao Liu, Hongwei Yang, Guoqing Xie. It tackles how to make LLM agents use workflows much better without having to manually design every single thing from scratch for every new task.

Jane: Right. So the core idea is that instead of designing one perfect workflow for a specific job, you build a whole pool of good candidate workflows first, and then the system learns which one to pick based on what the user actually asks. Is that right?

Tom: Exactly. It decouples workflow generation from selection. The authors say manually designing these workflows is really labor-intensive and it doesn't give the agent enough flexibility for different kinds of queries > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Lu: What I find interesting is how they build that initial pool using execution-guided evolution. It starts with an "Initial Generation" of a workflow for a seed query, then it uses "Execution-Guided Evolution" to refine it based on feedback > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Meng: From an engineering standpoint, I always worry about the cost of this feedback loop. The paper mentions that existing methods incur expensive API costs during the training phase feedback collection > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Lalam: I see how they address the cost issue by creating a workflow pool rather than optimizing one single workflow repeatedly, which should make the training phase feedback collection more efficient > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Jane: So they move away from optimizing a single task and towards having a diverse set of options ready to go > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. That’s the big shift we need to understand.

Tom: The second stage is where it gets clever. They formulate workflow selection as a Contextual Bandit problem > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. This lets the agent learn a policy tailored to the specific query it's seeing right now.

Title and authors: Lu: And they use this Contextual Bandit framework with a Hybrid Neural-Linear UCB selector > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. That combines neural reward prediction with a linear exploration bonus to keep things stable when the outcomes are uncertain.

Meng: I like the idea of balancing performance and sample efficiency that they mentioned > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. But what does that actually mean in terms of deployment?

Jane: It means the system learns when to trust a new workflow versus sticking with something it already knows is good, based on the context of the question > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. It’s adaptive routing for workflows.

Tom: And during inference, the agent just picks the workflow with the highest predicted reward from that learned policy > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. It’s a greedy selection based on what it has learned.

Lu: The results show this works really well across different benchmarks, even generalizing to Llama-three point one-70B backbones > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. That cross-domain transferability is pretty solid.

Meng: It’s good that it generalizes, but we need to talk about the overhead. The paper notes that this approach reduces training-phase API token overhead by no less than forty point five five percent and total end-to-end training token overhead by no less than twenty-nine point seven three percent on the MATH dataset > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. That’s a significant cost saving for us if we can actually deploy it reliably.

Jane: It sounds like they are proving that this method is not just theoretically sound, but practically more efficient than the manual design approach > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1. That’s what we need to see in production systems.

Tom: And the ablation study confirms that both the evolutionary pool construction and that hybrid UCB exploration actually contribute to those performance gains > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2. It means we can't just throw away either part of the framework without losing the benefit.

Title and authors: Lu: I also liked that they used Wilson score confidence intervals during inference to filter out unreliable workflows, requiring only the top fifty percent of workflows ranked by that metric > Improvements four.

Meng: Filtering things based on confidence is smart for stability when we’re running at scale > Improvements four. It cuts down on running potentially broken or very weak workflows during a real query.

Jane: So it’s not just about picking the best one, it’s about filtering out the ones that are statistically likely to fail even if they score high initially > Improvements four. That adds a layer of safety to the selection process.

Tom: The paper also showed that excluding pool generation cost reduces training token cost by no less than twenty-nine point seven three percent compared to baselines reporting those costs on MATH > Ablation and Cost Efficiency. That shows the value of having that initial pool construction part, even if it adds some upfront cost.

Lu: The generalizability is also strong, with scores averaging sixty-four point two four percent in Llama-three point one-70B experiments on both MATH and HumanEval benchmarks > Generalizability. It performs better than the Vanilla baseline by a large margin and is slightly better than MaAS there too.

Meng: I’m still skeptical about those benchmark scores because we haven't run them on real traffic, but the performance numbers are definitely compelling for a model like this > Generalizability. But the cost savings are what get my attention for production readiness.

Jane: So to wrap up, CB-Orchestrator is a two-stage framework that builds a diverse pool via evolutionary search and then uses contextual bandits with Hybrid Neural-Linear UCB for selection > Conclusion. It really moves us toward deploying versatile, cost-efficient LLM agents across different applications.

Tom: Yeah. It’s about making the workflow optimization process adaptive and sample efficient instead of rigid or just computationally expensive > Conclusion.

Lu: I think what this paper shows is that decoupling generation from selection is a very practical path forward for building more robust agentic systems > Conclusion.

Meng: For production, the cost efficiency metrics are what matter most right now when we’re talking about scaling these things up > Conclusion.

Lalam: CB-Orchestrator offers a concrete way to move away from manual workflow design toward automated, adaptive optimization for LLM agents > Conclusion.

The paper's summary: Tom: So we’re looking at how CB-Orchestrator works in a nutshell today. It’s this two-stage framework that stops you from having to manually design every single workflow for an AI agent > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Jane: Right. So, it builds a big pool of candidate workflows first, and then it learns how to pick the best one based on what the user asks > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: Exactly. Stage one is all about building that pool through execution guided evolution, which means it starts with a few initial ideas and then uses the actual feedback from running them to refine those workflows > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Jane: And stage two is where the magic happens. It treats choosing which workflow to use like a contextual bandit problem, learning a policy that adapts to every new query > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: The authors use this with something called the hybrid neural linear UCB selector, which mixes predicting rewards with a little exploration bonus to keep things stable when the outcomes are uncertain > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Jane: So, what does this mean for us? It means instead of having one rigid plan, the AI agent can dynamically switch strategies depending on the question it gets > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: Yeah. It lets the system learn which workflow is best for a specific task, which is way more flexible than just having one pre-made solution > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Jane: And it’s not just about finding *a* good workflow. They actually found ways to filter out the bad ones during the process using something called Wilson score confidence intervals > Improvements four. That helps keep the system from picking a workflow that might look good but is actually unreliable > Improvements four.

Tom: Those filters are key for deployment, because you don't want your agent failing on a real user query because it picked a fluke > Improvement four. Plus, they showed that even when you skip the initial pool building cost, you still get major savings in training tokens compared to older methods > Ablation and Cost Efficiency.

Jane: So the paper is proving that this adaptive selection process is more sample efficient than just trying to optimize one single workflow for everything > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: It’s about getting the best performance without having to run a massive search space for every new interaction, which is how you handle real-world scaling > Conclusion.

Jane: It points toward a future where AI agents aren't just following one set of instructions but are constantly optimizing their approach in real time based on the context they receive > Conclusion.

Tom: If this framework holds up across different backbones, like Llama models, it could actually make deploying complex agent systems much more practical for people building things now > Generalizability.

Jane: We'll look at those results from the Llama-three-70B experiments next, and see just how robust this works when you move to bigger models > Generalizability.

The paper's improvements: Tom: So we’re looking at what the authors suggest for improving CB-Orchestrator, moving past just making it work > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Jane: Right. So they’re talking about things like adding a way to detect when the agent is missing a whole strategy in its current workflow pool > Improvements four. It’s about checking for gaps in what it knows, and if it finds one, generating a new workflow on the fly > Improvements four.

Tom: That gap detection thing is interesting because it means the system doesn't have to wait for a massive retraining cycle just because the query is totally new > Improvement four. It lets you keep capability going during distribution shifts.

Jane: And they also mentioned using Wilson score confidence intervals during inference, not just picking the highest scoring workflow but filtering out anything that’s statistically shaky > Improvements four. They only pick the top fifty percent of workflows ranked by that metric > Improvements four.

Tom: That sounds like it adds a layer of safety to the selection process when you’re actually running stuff at scale > Improvement four. You don't want the AI picking something that looks good but is just fragile on a real user query.

Jane: So, they are pushing for this kind of online learning mode where the system actively expands its own toolbox based on what it sees > Improvements four. It moves it from a static pool to something that’s constantly evolving > Improvements four.

Tom: I like that idea of proactive expansion over just reacting when a failure happens later, which is how you build production systems > Improvement four. That proactive part is what separates this from older methods where you had to design everything upfront.

Jane: And they’re also exploring how to make the selection even more adaptive by looking at different ways to balance exploration and exploitation in that bandit problem > Contextual Bandit Selection. They are trying to find a better way for the context embedding and the reward prediction part > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: Yeah, getting that exploration bonus right is crucial because it determines how quickly the system finds a good new workflow without just repeating what it already knows > Improvement four.

Jane: And they’re also focusing on making the whole process more sample efficient overall, which I think is the big win for any AI agent research > Contextual Bandit Selection. It means getting better performance with less training data and less computation > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Tom: Exactly. If you can get that efficiency up, you can run these agents on much cheaper infrastructure > Ablation and Cost Efficiency. That’s where the real impact for engineers is > Conclusion.

Jane: So, the direction is clear: build a robust pool through evolution, learn to pick contextually with bandits, and add mechanisms to detect missing strategies and filter out risky options > Contextual Bandit Selection. It’s a lot of moving parts, but it’s all aimed at making these agents far more versatile > Conclusion.

Conclusion: Tom: So to wrap up, CB-Orchestrator is this two-stage framework that separates workflow building from how we select which one to use > Conclusion. It builds a diverse pool through evolutionary search and then uses contextual bandits with hybrid neural linear UCB for selection > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg1.

Jane: Right. It’s all about making the workflow optimization process adaptive instead of rigid or just computationally expensive > Conclusion. It decouples generation from selection > Preprint CB-ORCHESTRATOR: ADAPTIVE WORKFLOW OPTIMIZATION FOR LLM AGENTS VIA CONTEXTUAL BANDITS #pg2.

Tom: The real win here is that it makes agents more versatile and sample efficient for real-world use > Conclusion. It’s about having a practical path toward deploying these AI agents across different applications > Conclusion.

Jane: I think this suggests we are moving toward agent systems that can dynamically optimize their own internal logic based on the specific task at hand > Conclusion. That flexibility is what matters for real-world deployment.

Lu: I think the core idea here is decoupling those two steps, because if you can build a robust pool and then have a smart selection policy, you gain control over the system's behavior > Conclusion. It’s about giving the AI more strategic choice.

Meng: From my side, what I care about most is that it cuts down on the training token overhead by nearly thirty percent on benchmarks like MATH > Ablation and Cost Efficiency. That cost efficiency is a huge deal when you’re trying to deploy models in production > Conclusion.

Lalam: I'm confident in this direction because the results show strong cross-domain transferability, even outperforming some baselines on Llama-three-70B benchmarks > Generalizability. It seems like a really solid path forward for creating more reliable agents > Conclusion.

Jiahang Sun, Zhiwei Shang, Zhipiao Liu, Hongwei Yang, Guoqing Xie, Shuang Qiu, Zhongxiang Dai

The Chinese University of Hong Kong, Shenzhen · City University of Hong Kong

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

Comments: Presented at ICML 2026 Workshop on Compositional Learning: Safety, Interpretability, and Agents

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist Large Language Model (LLM) agents have demonstrated remarkable capabilities in tackling complex tasks through agentic workflows, but manually designing these workflows is labor-intensive and

Key concepts

Workflow Pool Construction (Stage 1): This stage automatically creates a large, diverse set of candidate workflows. It starts with an LLM generating initial ideas, then evolves them based on actual execution feedback from queries, and finally transfers successful workflow ideas between different queries to ensure the pool covers many possibilities.
Contextual Bandit Selection (Stage 2): This stage treats selecting a workflow as a learning problem. It uses the query's embedding as context to decide which workflow (arm) to use. A Hybrid NeuralLinear UCB selector balances exploring new workflows and exploiting known good ones, making the selection process very sample-efficient.
Evolutionary Search: This technique is used in Stage 1 to automatically discover high-quality workflows. It involves iteratively improving candidate workflows by testing them on queries and using the results (success or failure) to guide the evolution toward more effective structures, ensuring a broad range of useful options are generated.
Hybrid NeuralLinear UCB: This is the specific algorithm used in Stage 2 for selection. It combines a neural network prediction of reward with a linear bonus for exploration. This hybrid method helps the system efficiently decide which workflow to use by balancing predicted performance with the need to discover better options.

Terminology

Contextual Bandits — 1× in this paper · explained in 3 episodes

Summary

The gist Large Language Model (LLM) agents have demonstrated remarkable capabilities in tackling complex tasks through agentic workflows, but manually designing these workflows is labor-intensive and lacks flexibility to adapt to diverse queries.

CB-Orchestrator Framework

CB-Orchestrator is an adaptive framework that decouples workflow generation from selection. It proceeds in two stages: Stage 1 constructs a diverse workflow pool automatically from the training set via execution-guided evolutionary search, and Stage 2 learns a query-conditioned selecting policy by formulating workflow selection as a contextual bandit. This approach addresses the limitations of task-level optimization which suffers from rigidity and query-level optimization which creates a bottleneck in sample complexity.

Workflow Pool Construction (Stage 1)

The first stage constructs a diverse workflow pool W as the arm set for bandit selection. This construction targets validity (∀w ∈ W, ∃q ∈ Qtrain: r(q, w) ≥ τ) and completeness (∀q ∈ Qtrain, ∃w ∈ W: r(q, w) ≥ τ). The pool can contain human-designed, domain-validated, or automatically discovered workflows. The process involves three steps given seed queries Qseed ⊆ Qtrain: Initial Generation where an LLM generates an initial workflow w(q)0 = GENERATE(q, O, E), Execution-Guided Evolution where the workflow is evolved using execution feedback: w(q)j+1 = EVOLVE(w(q)j, q, tracej, errorj), and Cross-Query Transfer where similar successful queries are retrieved using embedding similarity.

Contextual Bandit Selection (Stage 2)

Stage 2 treats the query embedding ϕ(q) ∈ R d as context, workflows w ∈ W represented by ϕ(w) ∈ R d as arms, and answer quality r ∈ [0, 1] as reward. The Hybrid NeuralLinear UCB selector combines neural reward prediction with a linear exploration bonus to balance performance and sample efficiency under uncertain outcomes. The final selection score is given by UCB(q, w) = fθ(x(q, w)) + α · u(x(q, w)). This hybrid design uses the original feature space for uncertainty to measure feature-space novelty rather than directly estimating Var[r x].

Inference and Performance

During inference, CB-Orchestrator executes the workflow with the highest predicted reward w∗ = arg max w∈W fθ(x(q, w)). Evaluations across five diverse benchmarks demonstrate that CB-Orchestrator consistently outperforms all baselines. Notably, on the MATH dataset, CB-Orchestrator achieves better performance than strong recent baselines while reducing training-phase API token overhead by no less than 40.55% and total end-to-end training (including workflow pool generation) token overhead by no less than 29.73%.

Ablation and Cost Efficiency

The ablation study confirms that both the evolutionary pool construction and the Hybrid UCB exploration contribute to performance gains. The HybridUCB exploration strategy performs better than all baseline selection strategies. Furthermore, on the MATH benchmark, excluding pool generation cost reduces training token cost by no less than 29.73% compared to baselines reporting training costs on MATH. The cumulative average reward provides a signal for when to stop training.

Generalizability

CB-Orchestrator exhibits strong cross-domain transferability and generalizes effectively to different LLM backbones. For example, in the Llama-3.1-70B backbone experiments on MATH and HumanEval benchmarks, CB-Orchestrator achieves an average score of 64.24%, outperforming the Vanilla baseline by a large margin and demonstrating an overall performance slightly better than MaAS.

Conclusion

CB-Orchestrator introduces a two-stage framework for adaptive workflow optimization for LLM agents. It decouples workflow construction from adaptive selection. Stage 1 builds a diverse workflow pool via execution-guided evolutionary search; Stage 2 casts selection as a contextual bandit and applies Hybrid Neural-Linear UCB for sample-efficient exploration over query-workflow pairs. This framework offers a practical path toward deploying versatile, cost-efficient LLM agents across diverse real-world applications.

--- Page 1 ---

Generalizability

CB-Orchestrator exhibits strong cross-domain transferability and generalizes effectively to different LLM backbones. For example, in the Llama-3.

Improvements for AI systems

Improved workflow construction via execution-guided evolution: The system constructs candidate workflows through Initial Generation followed by Execution-Guided Evolution, which allows for iterative refinement based on feedback, ensuring that workflows are not only syntactically correct but also strategically robust for a given task type.
Query-conditioned selection via Contextual Bandit: Workflow selection is formulated as a Contextual Bandit problem, enabling the model to learn a query-conditioned routing policy by balancing exploration and exploitation, which allows the agent to adaptively choose the optimal workflow tailored to each specific query.
Hybrid Neural-Linear UCB for sample efficiency: The use of a Hybrid Neural-Linear UCB selector combines neural reward prediction with a linear exploration bonus, which is argued to be a practical design choice that prioritizes stability and computational efficiency over theoretical optimality.
Enhanced robustness through Wilson score filtering: During inference, the system employs Wilson score confidence intervals to filter workflows, requiring only the top 50% of workflows ranked by this metric, which filters out unreliable workflows with insufficient trials or consistently poor performance.
Adaptive pool expansion via Gap Detection: The optional online learning mode detects a Gap(q) indicating the query requires a strategy absent from the current pool, leading to On-Demand Workflow Generation to expand the pool and maintain continuous capability against distribution shifts.