Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model

arXiv:2604.17739 · cs.LG, cs.CL · Submitted 2026-04-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model".

Jane: Here's the gist of the paper. The gist The proposed method, TRUSTEE, is a cost-friendly approach for training tool calling agents by fully simulating dynamic environments using a free 8B language model and an adaptive curriculum learning mechanism.

Jane: The process involves three stages for each training iteration: "The first generates tasks for this batch (Section 2.2.1).

Tom: The second performs simulated interactions to collect trajectories (Section 2.2.2).

Title and authors: Tom: So we're looking at this paper titled "Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model." Jane, what stands out to you about that title right off the bat?

Jane: It sounds really ambitious, Tom. It suggests they've figured out a way to make training for tool calling agents much more accessible than before. They're talking about using something as small as an 8B language model to create entire simulated environments on its own, without needing huge datasets or super expensive models.

Lu: Yeah, the core idea is that you don't need those massive, proprietary environments anymore. Instead of relying on pre-made data or complex systems built by big labs, you can simulate the whole thing using an open-source language model as the backbone for all those steps.

Meng: From a deployment standpoint, I'm curious about how feasible this is when we think about what we actually need to run in production. Can we really rely on just an 8B model for generating everything?

Lalam: Based on what I've seen, the paper claims this method uses an LM to handle task generation, user simulation, tool simulation, and trajectory evaluation together for training. It’s designed to be self-contained because it doesn't depend on existing training data or tasks.

The paper's summary: Tom: Okay, so what does the actual process look like? Jane, can you break down how they are using this 8B model to build these environments? What’s the sequence of events for a single training iteration?

Jane: Basically, in each step of training, they go through three main stages. First, they generate tasks based on a subset of tools. Then, they perform simulated interactions to collect some kind of trajectory data. And finally, they reward those trajectories and optimize the policy using something called GRPO for stability.

Lu: The task generation part is interesting because it samples a subset of tools based on the current task difficulty level. This difficulty level is what they call D n, and that size determines how many tools are picked from a large tool repository, which can come from various places like existing datasets or crawled tools from the Internet.

Meng: So, if you have some tasks to do, how does that difficulty setting actually translate into something tangible for the agent? Is it just picking more tools?

Lalam: The paper explains that the task generation uses a backbone LM,, where e zero is generated based on structural prompts G which depend on the sampled tools f and the current difficulty D n, generating something like (p, g, q, c).

The paper's improvements: Tom: Okay, let's talk about those adaptive curriculum learning mechanisms they mentioned. Jane, how does that mechanism actually control the difficulty of the tasks during training? What are they adjusting?

Jane: They use several dimensions to control difficulty dynamically. They look at things like the number of tools being used from a repository and the number of interaction turns in a task, which naturally makes it more complex.

Lu: And they also adjust how specific the system prompt is that guides the agent. The paper says they expect the agent to learn high-quality tool calling under a general prompt, and then gradually decrease that specificity during training.

Meng: That sounds like a way to manage complexity without having to manually engineer every single level of difficulty. What about the reward criteria? Is that where they really fine-tune it?

Lalam: Yes, the rewarding criteria are also adjusted. They use a verifier, which is simulated by the backbone LM, to evaluate trajectories and generate a final reward r based on five levels of criteria.

Conclusion: Tom: So, wrapping up this whole idea of TRUSTEE, what's the big picture here? What does this mean for how we train these agents going forward?

Jane: It suggests that training doesn't necessarily need massive amounts of real-world data or handcrafted environments. It shows that fully simulated environments can be a powerful approach to tool learning, and it’s achieved using just an open-source 8B LM as the backbone.

Lu: The theoretical foundation points to this being about the gap between what the agent observes and what the environment knows. The improvement they show, J = J(pi theta) - J(pi zero), comes from both a transfer gain from the backbone LM's margin and a structural gain from the simulation elements.

Meng: If this works as well as they claim against baselines that need proprietary models, it lowers the barrier significantly for building custom tool-using agents. It shifts the requirement away from needing expert-made environments.

Lalam: I agree with Lu. The ablation studies show that you can't remove key components like user intent or expected tool calls and still have good performance across all subsets.

Tom: So, to summarize the paper "Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model," they propose a cost-friendly method, TRUSTEE, using an 8B model to simulate tasks and environments dynamically.

Jane: It’s about making tool learning less reliant on existing high-end resources by showing that simulated environments can be effective even with a local open-source language model as the backbone.

Lu: It moves the conversation toward how we structure these simulations to ensure we capture all the necessary information for successful learning.

Meng: I still have some questions about scaling this up and making sure it actually works reliably when you push it beyond these initial benchmarks.

Lalam: The paper does note a limitation, which is that there are constraints on scalability because of the bounded capability of the simulation LM, and uncertainty remains whether LMs can simulate stateful tools sufficiently well.

Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Junqiang Zheng, Saiyong Yang, Yunfang Wu

National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University Department of Large Language Models at Tencent

cs.LG, cs.CL

Submitted: 2026-04-20

Updated: 2026-10-17

Comments: Accepted to NeurIPS 2026 (main track)

Code: https://github.com/qiancheng0/ToolRL6https:

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: The gist The proposed method, TRUSTEE, is a cost-friendly approach for training tool calling agents by fully simulating dynamic environments using a free 8B language model and an adaptive curriculum

Key concepts

TRUSTEE: A cost-friendly approach that trains tool-calling agents by fully simulating dynamic environments. It uses a free 8B language model as the backbone to generate tasks, simulate interactions, and optimize policies through an adaptive curriculum.
Adaptive Curriculum Learning: A mechanism that dynamically controls task difficulty across several dimensions. This includes adjusting the number of tools, interaction turns, system prompt specificity, and rewarding criteria to ensure a smooth learning process for the agent.
Simulated Interaction: The process where the agent interacts with a simulated environment by sampling trajectories. The simulation involves generating outputs based on whether the previous step was a tool call or a text response, mimicking real-world user and environment feedback.
Information Asymmetry: The core idea that learning occurs in the gap between what an agent observes and what the environment knows. TRUSTEE exploits this asymmetry to improve performance beyond simple teacher-student distillation.

Terminology

Curriculum Learning — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 26 episodes

Summary

The gist The proposed method, TRUSTEE, is a cost-friendly approach for training tool calling agents by fully simulating dynamic environments using a free 8B language model and an adaptive curriculum learning mechanism.

How it works

The process involves three stages for each training iteration: The first generates tasks for this batch (Section 2.2.1). The second performs simulated interactions to collect trajectories (Section 2.2.2). The last rewards the trajectories and optimizes the policy (Section 2.2.3)
Task Generation involves sampling a subset of tools and then performing task generation based on the backbone LM Φ, where e0 = (p, g, q, c) is the generated task including the user persona p, the user intent g (i.e., the ultimate goal of the task), the first user query to trigger an interaction q, and the expected tools c to be called to complete the task
Simulated Interaction involves sampling M interaction trajectories where at ∼ πθ(·τ), which will be appended to the trajectory: τ = τ ⊕ at
Tool simulation occurs when at is a tool call, where et ∼ PΦ(·τ, omegaT). Otherwise, if at is a textual response to the user, et ∼ PΦ(·τ, omegaU). The feedback from the environment will then be appended to the trajectory: τ = τ ⊕ et
Rewarding and Optimization involves calling a verifier simulated by the backbone LM Φ to evaluate the whole trajectory and generates a final reward with r ∼ RΦ(·τ, omegaV). The optimization uses GRPO [25] for optimization due to its stability and simplicity

Adaptive Curriculum Learning

The method incorporates an adaptive curriculum learning mechanism that takes advantage of the dynamically controllable nature of the simulated environments and tasks, which controls the difficulty of the tasks in various dimensions adaptively and ensures a smooth learning process for the tool calling agent

The task difficulty is controlled by several dimensions:

** Number of Tools The number of tools sampled from F directly determines the overall difficulty. The larger number of available tools for the agent, the harder for the agent to make use of tools properly and precisely **

** Number of Interaction Turns If a task requires more interaction turns, it is naturally more complex. Therefore, we use the number of expected turns, which will be passed to the LM Φ for task generation, to control the difficulty. Meanwhile, the maximum number of interaction turns also evolves dynamically with the expected turns to ensure efficient training **

** System Prompt We expect the agent to learn to perform high-quality tool calling under a general system prompt and make the specificity of the system prompt for the agent gradually decrease during training **

** User Typically, a less expert user makes the task harder due to a lower level of knowledge [26], and an ambiguous user query usually confuses the agent and requires it to ask for further clarification. Therefore, we design three kinds of user personas (expert, beginner, and novice) together with three ambiguity degrees (clear, somewhat ambiguous, and highly ambiguous). Then, we use this as an input to the LM Φ for task generation which generates a more specific description of the user persona p (which will be further passed to the user simulator) and a user query q based on the requirements **

** Rewarding Criteria A stricter evaluation criteria naturally makes it harder for the agent to obtain a high reward. We increase the difficulty by gradually taking more criteria into consideration when calling the verifier Φ, which gives a positive reward only when all the criteria are met. Concretely, we design five levels of criteria: meeting user intent, using expected tools correctly, no hallucination, no redundant trials, and efficiency **

Theoretical Foundation and Results

The essential engine of learning in TRUSTEE lies in the gap between the agent’s limited observations and the environment’s privileged knowledge (i.e., information asymmetry) The improvement of TRUSTEE goes beyond a simple teacher-student distillation (from Φ to πθ) but can come from the structural design of the environment This is formalized by Proposition B.1, which states that the improvement ∆J = J(πθ) − J(π0) can be decomposed into the transfer gain (from the backbone Φ’s margin) and the structural gain (from E and omega), where Gainstruct > 0 even when Φ is not superior to θ

Empirical results show that TRUSTEE outperforms baselines which require extra external resources in most cases, confirming that simulated environments with a local 8B LM as the backbone could set a strong baseline for tool learning Specifically, for QWEN3-8B, TRUSTEE secures the highest scores in 5 out of 6 subsets and brings consistent improvements on all the subsets

Ablation Study Insights

Ablation studies confirm the necessity of various components. For example, removing any one of the components in the information set—user intent g, user persona p, or expected tool calls c—hurts performance on all subsets because Without user intent, the environment backbone does not have sufficient information regarding the latent ground truth and may not provide reliable simulation and verification Similarly, removing any dimension from curriculum learning results in performance degradation because without a gradual learning process, the overall performance of LMs downgrades Furthermore, a randomized difficulty prevents the agent LM from overfitting to undiversified tasks and environments and yields better performance during training

Conclusion

TRUSTEE demonstrates that simulated environments may be a powerful approach to tool learning that can even beat realistic data or synthesized verifiable environments This work moves a step towards democratized tool learning by showing that tool learning does not necessarily require established data, realistic human interactions, executable tools, or verifiable environments from human experts or advanced proprietary LMs By using a free 8B LM as the backbone, the paper suggests that simulated environments may be a powerful approach to tool learning that can even beat realistic data or synthesized verifiable environments

The limitations noted include constraints on scalability due to the bounded capability of the simulation LM, and uncertainty regarding whether LMs can simulate stateful tools sufficiently well. The broader impact is democratizing tool learning by lowering the barrier for researchers to effectively and efficiently train tool calling agents. The work concludes that with a proper design, fully simulated environments can effectively train tool calling agents even with a local open-source 8B LM as the backbone while requiring no annotated data or tasks

--- Page 26 ---