PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents

The gist: PACEvolve++ introduces an advisor-model reinforcement learning framework for test-time policy adaptation in evolutionary search agents, which decouples strategic search decisions from implementation to stabilize training during evolutionary search

Advisor Model and Decoupling

PACEvolve++ introduces a dedicated advisor model to make search-specific policy adaptation explicit The advisor learns the strategic parts of evolutionary search, including hypothesis generation, novelty assessment, and mutation selection while a stronger frontier implementation model translates the selected hypothesis into executable code. This design departs from standard evolutionary coding frameworks where the same model is used to both decide what to try and implement the resulting mutation. The framework separates what to try from how to implement it, aligning with post-training practices where reasoning and coding are developed as distinct capabilities before composition.

Phase-Adaptive Policy Optimization

The framework employs a phase-adaptive approach to train the advisor under non-stationary feedback, adapting its optimization strategy to different phases of the evolutionary process. Early in evolution, it uses group-relative feedback to learn broad search preferences, while later, as reward gaps compress, it emphasizes best-of-k frontier contribution to support stable refinement. This strategy aligns training with the log-diminishing reward structure of evolutionary search dynamics, stabilizing late-stage training while avoiding early-stage exploitation.

Search Dynamics Aware Objective

The method designs the search-dynamics aware objective to change the source of credit assignment as the search moves from exploration to frontier refinement. In the exploratory regime, it uses a raw group-relative baseline for dense within-group credit assignment, whereas in the refinement regime, it uses a pass@k-based marginal-contribution signal (PKPO) to credit candidates based on their contribution to the evolving best-of-k frontier. The phase schedule changes which signal dominates rather than merely changing the overall update magnitude.

Empirical Performance and Stability

Empirically, PACEvolve++ demonstrates strong performance across a range of real-world research and engineering tasks including expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation. The results show that PACEvolve++ consistently provides the strongest final reward while optimizing smoothly and converging the fastest across the evaluated tasks. Furthermore, auxiliary metrics clarify why baselines stall by showing that PACEvolve++ keeps gradient norms in a comparatively narrow band around 1 while maintaining materially higher entropy than Max@k, leading to smoother dynamics.

Task Complexity and Configuration

The experiments were conducted on small open-weight models (with 4B to 8B parameters) because end-to-end evolution often results in low success rates in implementation correctness. This model capacity concern motivated the design to separate idea generation from code implementation. The training configuration involves an online on-policy setting where each evolutionary search thread generates a candidate, which is then used for training in the same step. The evaluation settings vary by task, such as a 600s timeout for EPLB and a 1,200s budget for KuaiRec.

Disaggregated Results

Disaggregated metrics show that PACEvolve++ achieves superior performance compared to baselines across all tested tasks. For instance, in EPLB, PACEevolve++ achieved a Bal. Speed of 0.510 against Max@k's 0.458. In Protein Fitness Extrapolation, it showed a Pearson r of 0.733 compared to Max@k's 0.697. These results indicate that the method achieves faster convergence to better results across diverse engineering and research tasks.

Limitations

Due to the high costs of both RL training and evolutionary search, PACEvolve++ could not repeat experiments or run them over a longer horizon. Future work is left to scale up experiments or train models with stronger coding capabilities that may handle evolution end-to-end. The paper concludes by highlighting the importance of improving the reasoning policy, rather than just the search scaffold, to scale self-evolving agents to realistic domains.

1. Bold header: Phase-adaptive credit assignment for RL training. The system can dynamically shift its learning focus from broad search preferences in early exploration to frontier-contribution feedback during refinement, aligning policy learning with the log-diminishing reward structure of evolutionary search dynamics [8].

2. Bold header: Decoupled advisor-model architecture for task specialization. By separating the strategic decision-maker (advisor) from the implementation model, the system can learn search-specific policy adaptation explicit[ly] while leveraging a stronger frontier implementation model translates selected hypotheses into executable code [39], mitigating failures arising from complex codebases, integration details, and system constraints [46].

3. Bold header: Robust credit assignment under reward compression. The framework utilizes the phase-adaptive mixture of standardized group-relative and SLOO branches to ensure that in late-stage evolution where rewards compress, the objective shifts to credit assignment based on whether a candidate contributes to the evolving best-of-k frontier, preserving ordering across different reward scales.

4. Bold header: Improved convergence stability during training. The resulting system exhibits smoother dynamics and avoids pathologies observed in baselines, such as repeated late-stage gradient-norm spikes, leading to a more robust optimization trajectory across complex tasks like Expert-parallel load balancing and protein fitness extrapolation.

Large language models have become drivers of evolutionary search, but most systems rely on a fixed, prompt-elicited policy to sample next candidates. This limits adaptation in practical engineering and research tasks, where evaluations are expensive, and progress depends on learning task-specific search dynamics. We introduce PACEvolve++, an advisor-model reinforcement learning framework for test-time policy adaptation in evolutionary search agents. PACEvolve++ decouples strategic search decisions from implementation: a trainable advisor generates, assesses, and selects hypotheses, while a stronger frontier model translates selected hypotheses into executable candidates. To train the advisor under non-stationary feedback, we propose a phase-adaptive approach that adapts its optimization strategy to different phases of the evolutionary process. Early in evolution, it uses group-relative feedback to learn broad search preferences; later, as reward gaps compress, it emphasizes best-of- k frontier contribution to support stable refinement. Across expert-parallel load balancing, sequential recommendation, and protein fitness extrapolation, PACEvolve++ outperforms the state-of-the-art evolutionary search framework with frontier models, achieving faster convergence and stabilizing test-time training during evolutionary search.