TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

TIDE RL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

Abstract

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TIDE RL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, RA2P selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TIDE RL improves RL training goodput by up to 5.6× over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58×, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Introduction

Large Language Models (LLMs) are rapidly evolving from static text generators into autonomous agents capable of solving complex, real-world problems. In these agentic workloads, models do not merely produce a single-turn answer; instead, they iteratively interact with external, distributed environments, such as web browsers, smartphones, or computer OSes. This introduces a multi-turn execution flow. An agent generates an action, transmits it over the network to the environment, waits for the execution results, and resumes generation based on the newly returned observations. Because LLM outputs are non-deterministic, execution trajectories of the same task may differ drastically in turn count, context length, and wall-clock duration, producing workloads with extreme variance across both spatial and temporal dimensions.

To enhance these autonomous capabilities, researchers turn to reinforcement learning (RL). While RL has shown immense success in single-turn reasoning tasks, applying it to multi-turn agentic scenarios requires orchestrating massive, networked machine learning clusters. A typical RL workflow goes through four phases. In the rollout phase, a batch of agentic tasks runs concurrently, requesting actions from rollout workers and observations from environment workers to generate interactive trajectories in multiple turns. Later, the RL system evaluates these trajectories in the reward and reference phase. Finally, the system uses the output from the reward and the reference to update the model weights in the actor phase.

However, a small number of long-tail tasks in the rollout phase may block the pipeline and leave expensive GPU resources underutilized. For agentic RL, the right objective is goodput, which we measure through training throughput: the rollout'd tokens that reach the Trainer and advance an update counts. GPU waiting and repeated prefill recomputation are pure overhead. To maximize goodput, asynchronous systems separate these components onto different GPUs, unlike synchronous systems like VeRL that force rollout-train barriers. On a subset of GPUs, rollout worker groups perform continuous generation as producers. On the remaining GPUs, trainer worker groups, which colocate the reference and actor models, pull data from the global buffer to compute weight updates, as consumers. This decoupling theoretically allows generation and training to proceed concurrently, so that the rollout phase of long-tail tasks can take place simultaneously with the training phase of shorter tasks.

While effective for single-turn tasks, current asynchronous architectures exhibit severe systemic inefficiencies when processing multi-turn agentic workloads characterized by high concurrency and variable context lengths. We identify three critical challenges:

• KV cache preemption during intermittent interactions. Standard inference engines rely on request-level continuous batching, assuming each request is self-contained. However, multi-turn tasks proceed as a sequence of dependent requests separated by idle periods while waiting for external feedback. During these pauses, the engine greedily evicts its KV cache (the memory footprint of historical tokens) to serve requests from other concurrent tasks. Upon resumption, the system suffers a catastrophic cache miss, forcing redundant and computationally expensive recomputation (prefill) for an increasingly long historical context.

• Stalls and thrashing in reference inference. Existing systems share GPU resources between actor and reference workers within the trainer, enforcing strict stage dependencies. To avoid memory exhaustion, they either wait for large global batches to arrive before initiating updates, or frequently alternate between reference and actor models (which share the same architecture but have different weights) on the same GPUs for each micro-batch. The former creates massive computation bubbles that leave GPUs idle for up to 81% of the step time, while the latter introduces severe model-swapping thrashing overhead.

• Rigidity of resource allocation under shifting bottlenecks. Agentic RL operates as a producer-consumer pipeline. Because task complexity varies drastically, the system bottleneck dynamically shifts. A batch of complex tasks stalls the trainer as it waits for slow rollout data, while simple tasks create a large ready backlog at the beginning of a training step and then keep feeding new micro-batches rapidly. Relying on static resource partitioning between rollout and trainer worker groups, existing systems cannot react to either the initial ready backlog or the steady ready interval, structurally capping global throughput during these macroscopic workload shifts.

These intertwined challenges expose architectural flaws of existing systems, raising a critical research question: How can we build a unified, elastic training architecture that maximizes RL goodput, as measured by training throughput, under multi-turn agentic execution? The key is not to optimize rollout, training, and resource allocation as isolated modules. Multi-turn RL needs a shared readiness view: Rollout must preserve the task state that will become useful soon, Trainer must choose a ref-actor execution mode that matches how ready data arrives, and the cluster scheduler must move GPUs toward the side that removes the current goodput loss. Answering this requires co-designing the multi-turn memory management, distributed execution pipeline, and dynamic cluster resource allocation.

To this end, we identify two architectural blind spots in existing frameworks. First, systems such as AReaL and StreamRL expose asynchrony, but still treat the rollout-trainer split as a mostly fixed operating point; they cannot use the current ready backlog and arrival pace to jointly choose the ref-actor execution strategy and the GPU split. Second, standard inference engines allocate memory at the request level. Memory management must be elevated to the task level to gain global visibility over the multi-turn lifecycle, actively coordinating context pinning across distributed rollout ranks. Together, these changes make individual data parallel (DP) ranks schedulable resources rather than static members of a worker group.

Motivated by these insights, we present TIDE RL, an elastic and asynchronous distributed system tailored for multi-turn agentic RL goodput. TIDE RL solves these challenges through three co-designed components:

• Continuous Task Batching (CTB) replaces request-level continuous batching on each rollout rank with task-level admission, preemption, and resumption. CTB tracks per-rank token footprints in real time and applies a semantic-aware priority hierarchy—prioritizing evaluation boundaries, near-complete GRPO groups, active trajectories, and long contexts. This keeps useful rollout state resident until it can feed the global buffer, providing the stable producer side that RA2P and ERS rely on.

• Resource-Aware Ref-Actor Pipelining (RA2P) resolves the stall-thrashing dilemma through two complementary execution modes selected by two data-readiness signals: Ready-at-Start (RAS), the amount of micro-batch work already present when a training step begins, and Time Per Ready Micro-batch (TPRM), the interval at which additional micro-batches become available. In decoupled mode, Ref and Actor run on separate GPUs in a streaming micro-batch pipeline; a restructured computation graph defers loss computation into the backward pass, enabling Ref forward, Actor forward, and Actor backward to overlap with zero swap overhead. In colocated mode, RA2P uses ready-batch aggregation and zero-copy shared-memory transfer to amortize unavoidable swapping across ready bursts. Together, RA2P turns readiness into goodput under both startup backlog pressure and steady per-micro-batch arrival pressure.

• Elastic Resource Scaling (ERS) exploits RA2P's dual-mode structure to migrate individual ranks between rollout and training on the fly. ERS reads RAS and TPRM from the global buffer: a small RAS with slow TPRM indicates rollout starvation and triggers colocated RA2P plus more Rollout ranks, whereas a large RAS or fast TPRM indicates trainer-side pressure and triggers decoupled RA2P plus more Trainer capacity. Crucially, ERS piggybacks weight migration onto the natural cache-flush boundary that on-policy RL already mandates at every sync, achieving zero-overhead elasticity: no pipeline suspension, no NCCL group rebuild, no wasted KV cache.

We implement TIDE RL with a highly modular architecture, ensuring seamless compatibility with diverse multi-modal and text-only tasks, and easy integration with mainstream serving and training backends. We deploy the system on a 4-node cluster with 32 NVIDIA H100 GPUs and evaluate models with different architectures. Our evaluations demonstrate large performance gains over existing RL frameworks. End-to-end, TIDE RL raises RL training goodput, measured by training throughput, by 1.8–7.0× on text-only tasks (reducing time-to-convergence by 51.1%) and improves multi-modal goodput by over 33%, reducing training time by 62.2% with similar performance. Component studies close the loop from motivation to design: CTB improves generation goodput by preserving task state, RA2P reduces per-step training time by up to 44.3% under different readiness patterns, and ERS cuts total idle waiting time by 87.4% by scheduling RA2P modes and GPU ranks from the same readiness signals.

Background and Motivation

Agentic RL workloads in this paper follow Group Relative Policy Optimization (GRPO), whose execution alternates between trajectory generation and model update. The agentic GRPO pipeline consists of four sequential stages to train a new model A from the original model O:

• Rollout Generation: Rollout hosts A to generate actions. Crucially, this involves multi-turn interactions: the model generates an action, pauses to wait for environment execution, receives the observation, and generates the next action. For a given task, a group of G distinct multi-turn trajectories is sampled. Each trajectory may comprise different turns and lengths of messages.

• Reward Calculation: Once the trajectories in a group are completed, the environment evaluates them. GRPO determines the advantage Ai by standardizing the rewards ri strictly based on the performance of other trajectories within the same group: Ai =(ri −mean(r))/std(r).

• Reference Calculation: Ref computes the base logprobabilities on O of generated trajectories. This provides a Kullback-Leibler (KL) divergence penalty to prevent the policy from over-optimizing the reward.

• Actor Update: Actor executes the forward and backward passes on A to update its weights (i.e., update the policy). Once updated, the latest weights are synchronized back to Rollout for the next iteration.

The Trainer comprises Ref, Actor and the reward calculation, and consumes the trajectories from Rollout. Traditional synchronous architectures enforce a strict alternation between the rollout and training stages. This coupling causes severe GPU idling, as the training must wait for the long-tail task to complete. To break this resource coupling, modern systems typically adopt an asynchronous execution paradigm. They decouple generation and training into physically independent workers. This allows Rollout to continuously generate new groups of responses while Trainer consumes historical data in the background, significantly boosting hardware throughput. However, this asynchrony introduces policy staleness, as Rollout generates data using older model weights. Since algorithms like GRPO are fundamentally on-policy, excessive off-policy data severely degrade training stability. Consequently, asynchronous systems must strictly bound this staleness gap, requiring timely weight synchronization to prevent performance degradation.

Key Challenges in Asynchronous Agentic RL

Although asynchronous RL frameworks overlap generation and training, their execution traces under multi-turn agentic workloads reveal three ways the system loses RL goodput.

C1: Catastrophic KV Cache Preemption in Rollouts. Monitoring the KV cache hit rate and generation throughput during a single training step reveals a typical three-phase fluctuation rather than stable cache utilization. At the beginning of the step, the hit rate increases as a large influx of new short sequences is initially cached. However, the hit rate subsequently plummets and remains at only ∼12%, dragging the throughput down to ∼650 tokens/s·instance. A spike in the hit rate occurs at the very end of the training step. The fundamental flaw lies in the mismatch between request-level scheduling and multi-turn semantics. When waiting for environment feedback, the underlying inference engine, driven by standard continuous batching to maximize concurrency, schedules new requests, thereby preempting the tasks' KV caches. The engine suffers massive cache misses as the observation returns, and must recompute the rapidly growing historical context. The late recovery in hit rate is merely an artifact of dropping concurrency: as the global batch nears completion, no new tasks are admitted, allowing the few straggling tasks to safely retain their cache without preemption.

C2: Stalls and Thrashing in Training. Examining the Trainer's execution trace reveals a dilemma between computation stalls and I/O thrashing. Standard execution forces the Trainer to stall significantly while waiting for a full global batch to complete, a delay severely exacerbated by long-tail tasks. Conversely, if we attempt to mitigate this idle time by streaming micro-batches, the system falls into another trap: massive I/O thrashing that degrades training GPU goodput. This dilemma is structurally rooted in the memory-intensive nature of on-policy algorithms like GRPO. The Trainer must execute reference calculations and actor updates sequentially using two distinct models. Under standard execution, the system waits for all rollout tasks in a global batch to finish, making the Trainer's idle time bottlenecked by the slowest long-tail samples. To avoid these macroscopic stalls, the system could adopt a streaming mode to process micro-batches as soon as they arrive. However, existing systems (e.g., VeRL and AsyncFlow) cannot run both models on the same GPUs simultaneously due to the runtime memory consumed by training activations. Consequently, streaming micro-batches forces the system to frequently load and offload model weights between the CPU and GPU over PCIe, replacing the stall with a model-swapping bottleneck.

C3: Shifting Bottlenecks Invalidating Static Allocation. Tracking the Rollout Wait Time (RWT) and Training Wait Time (TWT) across 100 steps under various static Rollout GPU ratios reveals that the system bottleneck oscillates wildly. A lower rollout ratio (0.125) consistently starves the Trainer (high TWT), while a higher ratio (0.5) generates data too fast, causing Rollout to stall (high RWT). Most crucially, even…

Based on the paper, here are the specific improvements you can make to AI systems:

Improvement: Replace request-level continuous batching with task-level admission, preemption, and resumption that tracks token footprints per rank in real time.

What the improved system can do:

• Maintain KV cache for multi-turn tasks across idle periods instead of evicting them

• Prioritize tasks by semantic importance (evaluation boundaries > near-complete GRPO groups > active trajectories > long contexts)

• Prevent catastrophic cache misses that cause redundant prefill computation

• Achieve 1.58× higher KV cache hit rate and 1.15× higher generation throughput

Improvement: Dynamically select between two execution modes based on data readiness signals (RAS and TPRM):

• Decoupled mode: Run Ref and Actor on separate GPUs with deferred loss computation in backward pass

• Colocated mode: Use ready-batch aggregation and zero-copy shared-memory transfer

Improvement: Migrate GPU ranks between rollout and training on the fly using the same readiness signals, piggybacking weight migration onto natural cache-flush boundaries.

Improvement: Jointly optimize micro-batch formation and allocation across DP ranks with:

• Small final batches to minimize pipeline flush bubbles

• Zig-zag allocation for colocated mode (balance compute load)

• Smallest-last sequencing for decoupled mode (minimize tail latency)

Improvement: Dispatch tasks based on measured token usage rather than raw request counts, with constraints on environment concurrency, policy staleness, and task diversity.

Improvement: Resume paused tasks on ranks most likely to retain shared prefixes, with worker affinity based on group IDs.

Improvement: When Rollout is scaled down, Trainer adaptively reduces fetch size toward Bmin and retains a micro-batch reservoir.

Improvement: During cold starts or evaluation phases, maximize Rollout group size preemptively rather than waiting for readiness signals.

Improvement: Overlap PCIe weight-swapping with sync params broadcast, discarding stale weights and loading new models during synchronization windows.

Improvement: Quarantine failed nodes, exclude them from broadcasts, and use asynchronous distributed checkpointing.

Overall system capability: The improved AI system can achieve 5.6× higher RL training throughput on text-only tasks and 33%+ on multi-modal tasks compared to existing systems, while reaching similar task performance with 51.1% less training time.

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, RA 2 P selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6 times over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58 times, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.