Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds

arXiv:2610.14860 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds".

Jane: The gist The framework Terrarium synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper today, "Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds." Basically, the idea is that we can make general-purpose personal assistants by training them on data from real applications.

Jane: Right. The core problem they're tackling is that real apps are hard to train on at scale because you usually can't get access, and resetting their state is tough. So this framework aims to solve that by creating synthetic training data for these assistants.

Lu: Yeah, what Terrarium does is synthesize executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics from those worlds, and then it iteratively validates and refines both through agent rollouts.

Meng: It sounds like they are trying to build a system that creates these isolated environments where the AI can actually practice interacting with an app without needing live access to the production backend.

Lalam: Exactly. For App World synthesis, they organize public web information and in-application observations into structured app specifications that guide the construction of self-contained runtimes with callable operations and mutable, resettable state #pg2.

Tom: And for task synthesis, they build tasks grounded in those App World states together with evaluator-side outcome rubrics that assess success without prescribing reference trajectories #pg3. That’s a neat way to ground the training data.

Jane: So before we get into how they actually build these things, can you tell us more about what kind of information they are starting with for App World synthesis?

Tom: Well, the process starts with Application Information Collection where they use an LLM for web search and a GUI agent to record action traces, screenshots, and accessibility trees #pg3. They then use an LLM to consolidate that into a semantic application summary and normalize capabilities into workflow steps #pg3.

Lu: The goal of that specification construction is to derive model-visible operation contracts from page three which lets them build these self-contained App Worlds with app-specific state, callable operations, transition logic, and a resettable seed state #pg2 <ref:2610.14860#pg1>.

Meng: That sounds like a lot of work just to get the structure right. How do you handle the diversity between different applications when you're trying to build this common intermediate representation?

Lalam: The app specification supports a shared construction process across diverse applications while retaining app-specific workflows and constraints from page three which is key for making it scalable #pg2 <ref:2610.14860#pg1>.

Paper summary: Jane: And once they have these worlds, what’s the next step in generating the actual training tasks? How do they turn an executable App World into something a model can actually learn from?

Tom: Task synthesis builds on those worlds by grounding user intents in concrete App World states and pairing them with evaluator-side outcome rubrics #pg3. This leads to Intent Execution supervision and Intent Completion supervision #pg3.

Lu: The paper shows that combining Intent Execution and Intent Completion improves over just Intent Execution across every benchmark subset from page three which suggests that having both types of supervision is beneficial #pg3 <ref:2610.14860#pg1>.

Meng: If we're talking about deployment, how does the iterative validation process work? Do you actually run these agents against the generated worlds repeatedly?

Lalam: Yes, Terrarium validates and refines both through agent rollouts in a closed-loop construction process #pg2. This rollout feedback drives subsequent rounds of refinement and validation #pg2.

Jane: That sounds like a very thorough way to ensure the generated data is actually good enough for training a general assistant. What are some of the numbers they report on this training suite?

Tom: They constructed a training suite of two hundred seventy-five App Worlds and thirteen thousand three hundred forty-four tasks for outcome-based reinforcement learning across three Qwen3 model scales: 4B, 8B, and 14B parameters #pg2.

Lu: And the performance gains are pretty solid. Across those scales, models trained with Terrarium outperform the corresponding Qwen3 models by an average of eighteen point three four percentage points on the τ2-bench and ten point nine four percentage points on the AppWorld bench #pg1.

Meng: eighteen point three four for the tau squared-bench, that's a significant lift compared to just using a standard model trained without this specific app-grounded data <ref:2610.14860#pg1>. What about scaling up the models?

Tom: Scaling Terrarium from 4B to 14B improves the microaverage by five point four zero points on tau squared-bench and four point nine six points on AppWorld from page six #pg1. And as you add more data, say spanning from Page two to Page five both benchmark microaverages improve consistently #pg2 <ref:2610.14860#pg1>.

Jane: So, if a person listening here who doesn't read the paper is wondering what this means for them directly, what's the big picture implication?

Lu: It suggests that app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across broad application ecosystems #pg5. It moves us away from just training on generic text to training on actual interaction with software.

Paper summary: Meng: I worry about the diversity of these worlds, though. The median world seems pretty dense, combining seven capabilities and thirty-six documented workflow steps with nine entity types and fourteen agent-visible operations #pg4. How do you manage that kind of complexity?

Lalam: That complexity is what they are building for. The diversity of the synthesized App Worlds is substantial, with those worlds spanning eight categories, each accounting for six point nine percent to sixteen point four percent of the corpus #pg3.

Tom: It shows that you can build a very rich training corpus without it becoming unusable, as long as you stick to this synthesis pipeline #pg5. This framework is designed to create executable, resettable App Worlds and grounded training task packages with evaluator-side rubrics using agent rollouts for closed-loop validation and refinement from page two #pg2 <ref:2610.14860#pg1,training task packages with evaluator-side rubrics>.

Jane: It sounds like a very solid structure for building assistants that can actually *do* things across different software, not just talk about them. So what's the final word on this framework?

Lu: Terrarium provides a scalable path toward general-purpose assistants by offering effective supervision across model scales and benefiting from broader app-grounded training data #pg5. It’s implemented at scale to construct two hundred seventy-five App Worlds and thirteen thousand three hundred forty-four tasks from page five #pg2.

Meng: It seems like a way to generate high-quality, specific interaction data where traditional methods struggle due to access restrictions and state management issues #pg1. I'm interested in seeing how this translates when we try to deploy these agents in production environments where the application logic is really complex #pg3.

Lalam: The framework is designed to create executable, resettable App Worlds and grounded training task packages with evaluator-side rubrics using agent rollouts for closed-loop validation and refinement from page two #pg2 <ref:2610.14860#pg1,training task packages with evaluator-side rubrics>. This suggests a method for creating reliable interaction data without needing live production access #pg1.

Tom: That's the main point, right? We're not just training models on text; we're training them in an environment that actually mimics using an application, and this Terrarium method makes that scalable.

Jane: So the idea is that by making these structured worlds and tasks, we create a much more robust foundation for assistants interacting with the real world than we have right now.

Lu: Exactly. The results show app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across diverse applications #pg5. We'll be looking at how this opens up new research avenues for training agents in these specific application domains #pg5.

Conclusion: Tom: So, to wrap up this talk about Terrarium, we're looking at how they built a system for training general personal assistants using real application data by synthesizing these structured worlds and tasks on a massive scale.

Jane: Yeah, so essentially, what's the big picture here? The authors are showing that you can move beyond just training models on plain text to actually training them inside the context of how an application works.

Tom: Exactly. They took messy real-world app info and turned it into these executable App Worlds with resettable states, which is a huge step because it means the AI isn't just guessing; it’s interacting with something that has defined rules.

Jane: And the results on the numbers are pretty compelling. Models trained this way outperformed their standard Qwen3 counterparts by about eighteen point three four percentage points on one benchmark and ten point nine four on another.

Tom: That's a solid lift, and they show that even when you scale up the model size from four billion to fourteen billion parameters, the improvement keeps coming—it goes up by five point four for one test and nearly five points for the other.

Jane: What does that actually mean for someone who isn't working on these models every day? It suggests that this approach of creating specific training data grounded in actual app workflows is a very scalable way to build assistants that can handle a wide variety of software.

Tom: The authors are basically arguing that this method of synthesizing worlds and tasks provides the necessary supervision across different model sizes, which is something they found really important for getting good results consistently.

Jane: It points toward a future where general purpose assistants aren't just good at chatting; they’re actually capable of operating reliably within complex application ecosystems.

Tom: Right. And since they built this whole pipeline—from collecting web info to generating those thirteen thousand tasks—they’re open-sourcing it so other researchers can actually use this method to train their own agents.

Jane: It's a really interesting foundation they’ve laid there, and I think seeing how these structured worlds are used in the next set of experiments is going to be really insightful.

Mengzhou Wu, Yuzhe Guo, Jiawen Tao, Xiaokun Yuan, Yuan Cao, Gang Cao, Yuetang Deng, Zhi Gong

School of Computer Science, Peking University School of Computer Science Department Beijing Tongming Lake Information Technology Application Innovation Center WeChat Tencent Inc. Simon Fraser University Institute of Systems for Advanced Computing Fudan University

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Code: https://github.com/THUDM/slime

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The framework Terrarium synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics, and iteratively validates and

Key concepts

App World Synthesis

This process organizes public web data and in-app observations into structured specifications. These specifications guide the creation of self-contained runtimes for applications, ensuring they have callable operations and mutable, resettable state. This allows the framework to build realistic digital environments from real app information.

Task Synthesis

This involves grounding user intents within the constructed App World states and pairing them with evaluator-side outcome rubrics. This creates specific training tasks that assess success without needing pre-defined reference paths, helping agents learn how to achieve desired outcomes in a real application context.

Evaluator-Side Rubrics

These are criteria provided by human evaluators to assess the success of a generated task or agent action. They define what constitutes a successful outcome for an intent execution or completion, providing the necessary supervision for training models effectively without dictating exact steps.

Agent Rollouts

This is the iterative validation and refinement process where trained agents execute tasks within the synthesized App Worlds. The results from these rollouts are used to validate and improve both the App Worlds and the generated training tasks, creating a closed-loop system for continuous data improvement.

Summary

Improvements for AI systems

Sources

• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• Towards General Agentic Intelligence via Environment Scaling
• Scaling Synthetic Data Creation with 1,000,000,000 Personas
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Kimi K2: Open Agentic Intelligence
• GPT-4 Technical Report
• Gorilla: Large Language Model Connected with Massive APIs
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis
• RestGPT: Connecting Large Language Models with Real-World RESTful APIs
• AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
• UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics
• AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
• Qwen3 Technical Report

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks