OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

arXiv:2610.22200 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use".

Jane: As an excellent, fastidious, and diligent researcher, I have meticulously analyzed both provided excerpts concerning "OpenGCUA:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're looking at this paper called OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use. It sounds like they're trying to move beyond just using graphical interfaces and build something that works across both visual and programmatic ways on actual computers.

Jane: Right, so the idea is to create a generalist computer-use agent, or GCUA, where the model decides how it observes, acts, and organizes its work across these different kinds of interfaces > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1 <ref:2610.22200#pg2>.

Lu: What's interesting here is they are targeting systems on users' own computers, which brings up challenges like shared control and interacting with background windows that aren't usually in benchmark environments > Page three of that work reads: "These systems also target tasks on users’ own computers, where shared control, interaction with background windows, and evolving requests introduce challenges not fully captured by benchmarks" #pg3 <ref:2610.22200#pg2>.

Meng: I wonder if the authors really figured out how to make this robust for real-world deployment without it just breaking when something unexpected happens > Page two of that work reads: "Contents one Introduction three two Generalist CUA: Definition and Positioning" #pg2 <ref:2610.22200#pg2,Contents 1 Introduction 3 2 Generalist CUA: Definition and Positioning>.

Lalam: From what I'm seeing in the literature, the shift is moving from models just doing screenshot-based observation to things that can actually write code and use multiple tools > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" <ref:2610.22200#pg2>

Tom: Exactly. The core thing they're pushing is code execution as a main feature, which lets the model not only interact with GUIs but also write and run programs that mix GUI stuff with actual programming logic > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" <ref:2610.22200#pg2>

Jane: It sounds like they see the problem as being in the gap between what we can do with just visual feedback and what's needed for real computer tasks > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3 <ref:2610.22200#pg2>.

Title and authors: Lu: And they tackle this by introducing harness-to-model decision transfer, which lets the agent make its own workflow choices instead of just following fixed rules > Page two of that work reads: "three point two Harness-to-Model Decision Transfer" #pg2 <ref:2610.22200#pg2,3.2 Harness-to-Model Decision Transfer>.

Meng: So the model decides when to request a screenshot or how to organize its task state internally, rather than relying on a simple harness rule > Page one of that work reads: "three point two Harness-to-Model Decision Transfer" #pg2 <ref:2610.22200#pg2,3.2 Harness-to-Model Decision Transfer>.

Lalam: And they frame this decision making around three choices at every step: what to observe, how to act, and how to organize the work > Page two of that work reads: "three Agent Design: Code Execution and Harness-to-Model Decision Transfer" #pg2 <ref:2610.22200#pg2,3 Agent Design: Code Execution and Harness-to-Model Decision Transfer>.

Tom: That's the structure. But they're not just doing this abstract design; they are implementing it using tools like PyAutoGUI and bash in Python, with Playwright for the real device interactions > Page two of that work reads: "three point one Implementation" #pg2 <ref:2610.22200#pg2>.

Jane: And they've even developed a preliminary cross-device SDK to try and give this unified programming interface across different platforms like computers and mobiles > Page two of that work reads: "three point one Implementation" #pg2 <ref:2610.22200#pg2>.

Lu: A big point for me is the real-device harness they built, which seems designed to handle things like sharing machines with users or interacting with windows in the background, which are hard to get in standard benchmarks > Page two of that work reads: "three point one.three Limits of Visual-Only Feedback" #pg2 <ref:2610.22200#pg2,3.1.3 Limits of Visual-Only Feedback>.

Meng: I'm curious about the efficiency aspect, because running these kinds of agents on a real machine can be very expensive and slow > Page two of that work reads: "seven Efficiency as a First-Class Metric" #pg2 <ref:2610.22200#pg2,Efficiency as a First-Class Metric>.

Lalam: They found some pretty solid numbers when they tested this full OpenGCUA setup against baseline methods like GUI-only execution > Page one of that work reads: "Quantitative Gains" #pg1 <ref:2610.22200#pg2>.

Tom: They saw the mean partial score go up by nineteen point six points on the OSWorld-v2 set with the full GCUA setting compared to just using a GUI > Page one of that work reads: "Quantitative Gains" #pg1.

Jane: And they also found a big reduction in steps per task, cutting them by an average of fifty-three point zero percent on that same set > Page one of that work reads: "Quantitative Gains" #pg1.

Title and authors: Lu: The integration of code execution specifically added another six point one points to the mean partial score and cut the steps by about fourteen point eight percent compared to a GUI plus command line baseline > Page one of that work reads: "Code Execution Efficacy" #pg1.

Meng: That's promising, but I want to talk about the model capability part because those numbers don't tell me much about what the models can actually *do* > Page one of that work reads: "Model Capability Insights" #pg1 <ref:2610.22200#pg2>.

Lalam: They looked at four real macOS tasks and found a difference in reasoning efficiency between models under the real-device harness, specifically noting token usage differences > Page one of that work reads: "Model Capability Insights" #pg1 <ref:2610.22200#pg2>.

Tom: Wait, I read something about Qwen3 point 8-Max using nineteen point four times as many reasoning tokens as GPT-six Astra on four real macOS tasks, even though GPT-six Astra finished them in only thirty-four point five steps compared to Qwen3 point 8-Max's fifty point eight steps > Page one of that work reads: "Model Capability Insights" #pg1.

Jane: So it suggests that the real power of code execution isn't just writing executable code, but getting it applied correctly to cover the entire task scope and verify completion, not just being able to write loops and conditions > Page one of that work reads: "Model Capability Insights" #pg1 <ref:2610.22200#pg2>.

Lu: I think that points toward the fact that the system needs to be smart about when it decides what kind of interaction is best for a given moment > Page two of that work reads: "three Agent Design: Code Execution and Harness-to-Model Decision Transfer" #pg2 <ref:2610.22200#pg2,3 Agent Design: Code Execution and Harness-to-Model Decision Transfer>.

Meng: From my production side, I worry about deployment complexity—how hard is it to get this whole harness running reliably when you're dealing with actual user machines > Page one of that work reads: "three point one Implementation" #pg2 <ref:2610.22200#pg2>.

Lalam: The authors acknowledge that while they have developed a real-device harness, the work on real-world evaluation environments is still an area to develop further > Page two of that work reads: "four Real-World Evaluation" #pg2.

Tom: So, to wrap up the idea of OpenGCUA: it's about creating a system where models can dynamically choose between visual and programmatic interactions based on what the task demands > Page one of that work reads: "two Generalist CUA: Definition and Positioning" #pg2 <ref:2610.22200#pg2,2 Generalist CUA: Definition and Positioning>.

Jane: It’s about giving the model more autonomy in choosing its path, moving it away from just following a fixed set of rules in a harness > Page one of that work reads: "three point two Harness-to-Model Decision Transfer" #pg2 <ref:2610.22200#pg2,3.2 Harness-to-Model Decision Transfer>.

Title and authors: Lu: The main contribution is essentially creating this architecture—the code execution and the decision transfer mechanism—and then proving it works on real hardware > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" <ref:2610.22200#pg2>

Meng: For me, the main thing is that this architecture gives us a blueprint for how to build these generalist agents without getting completely stuck in a GUI-only loop > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" <ref:2610.22200#pg2>

Lalam: The OpenGCUA project itself is significant because they are making the full stack open, so the community can build on this instead of being locked into one specific implementation > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1 <ref:2610.22200#pg2>.

Tom: So, it’s a lot of moving parts: code execution, decision transfer, real-device testing, and making the whole stack open so everyone can contribute > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1 <ref:2610.22200#pg2>.

Jane: It’s really about taking the computer use question to a new place, focusing on how models can complete tasks using whatever means works on a real machine > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3 <ref:2610.22200#pg2>.

Lu: It sets the direction for future research by showing that we need these kinds of hybrid architectures if we want agents to be truly generalist in their computer use > Page one of that work reads: "two Generalist CUA: Definition and Positioning" #pg2 <ref:2610.22200#pg2,2 Generalist CUA: Definition and Positioning>.

Meng: I guess what this means for deployment is that we can start thinking about how to integrate code execution as a standard tool for building agent systems, not just an add-on > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" <ref:2610.22200#pg2>

Lalam: So, the OpenGCUA paper provides a concrete way forward for building agents that can handle the complexity of real computer use > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1 <ref:2610.22200#pg2>.

The paper's summary: Tom: So, this paper is OpenGCUA, and they’re building this whole system—this stack—to be open so everyone can build on it for generalist computer use > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Jane: Right, so what this means is they’re not just handing us one specific way to make an AI use a computer, but giving us the tools to build on it > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Lu: The main idea is this agent can switch between seeing things visually, like a screenshot, and using code, like running a Python script or a shell command > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Meng: And they’re making it really flexible by letting the model decide when to use what tool—a GUI action, an accessibility operation, a command line interface, or an API call > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Tom: They’re using something called harness-to-model decision transfer to let the agent make those choices autonomously instead of just following fixed rules > Page two of that work reads: "three point two Harness-to-Model Decision Transfer" #pg2.

Jane: So, the model gets its own autonomy over how it plans its steps, deciding when it needs a screenshot or how to track what it’s done internally > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Lu: It’s about giving the agent three big decisions at every step: what to observe, how to act, and how to organize its work > Page two of that work reads: "three Agent Design: Code Execution and Harness-to-Model Decision Transfer" #pg2.

Meng: And they're not just designing this in a vacuum; they’ve built a real device harness that tackles the messy stuff you actually deal with, like sharing machines or background windows > Page three of that work reads: "three point one.three Limits of Visual-Only Feedback" #pg2.

Tom: The evaluation numbers are pretty telling too; when they use this full configuration against just a GUI method, the mean partial score jumps by nineteen point six points on the OSWorld-v2 set > Page one of that work reads: "Quantitative Gains" #pg1.

Jane: And they cut down on the steps needed to finish a task by fifty-three percent on that same test set when they use this full system > Page one of that work reads: "Quantitative Gains" #pg1.

Lu: The integration of code execution specifically added another six point one points to the mean partial score and reduced steps by fourteen point eight percent against a GUI plus command line baseline > Page one of that work reads: "Code Execution Efficacy" #pg1.

Meng: That’s encouraging, but they also found some interesting things about how different models perform under this real-device test on macOS, like one model using nearly twenty times more reasoning tokens than another > Page one of that work reads: "Model Capability Insights" #pg1.

Tom: So it seems the real value isn't just writing code, but getting that code to apply correctly across the whole task scope and verifying completion > Page one of that work reads: "Model Capability Insights" #pg1.

Jane: It suggests that for an AI to be truly generalist in computer use, it needs this kind of hybrid architecture where it can dynamically choose between visual and programmatic interactions based on what the task demands > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3.

Lu: This paper really sets the direction for future research by showing that we need these hybrid architectures if we want agents to be truly generalist in their computer use > Page one of that work reads: "two Generalist CUA: Definition and Positioning" #pg2.

Meng: It means we should start thinking about how to integrate code execution as a standard tool for building agent systems, not just an add-on feature > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" #pg2.

The paper's improvements: Tom: So, moving past just the core idea, what are the specific improvements they’re pushing for in this OpenGCUA stack? > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Jane: They’re talking about making sure this whole system is truly modular, so people can plug in their own components without being locked into a single implementation > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Lu: They’re emphasizing this open nature, which is key for pushing this architecture further because it lets different teams focus on specific parts, like the code execution engine or the real device harness > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1.

Meng: From a deployment standpoint, that openness is good because if someone builds a better way to handle background windows on macOS, they can just drop it into the harness without breaking the whole thing > Page three of that work reads: "three point one.three Limits of Visual-Only Feedback" #pg2.

Tom: They’re also focusing on how we use those intermediate results—the stuff that happens during execution—to decide what to do next, which is a big mechanical step forward for the model > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" #pg1.

Jane: That means the AI isn't just following a script; it’s making decisions based on what happened in the last step, which is how you get that autonomy we talked about earlier > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Lu: They are also pushing the concept of unified cross-interface program space, meaning the agent can write one big program that handles everything—GUI actions, an accessibility operation, a shell command and an API call all together > Page two of that work reads: "Unified Cross-Interface Program Space" #pg2.

Meng: That sounds efficient if it’s actually runnable on a production machine because it bundles all those different interaction types into one execution block > Page one of that work reads: "Unified Cross-Interface Program Space" #pg2.

Tom: And they’re building this real-device harness to handle the weird, non-standard stuff that happens when you're using a computer with other people or when windows are hidden > Page three of that work reads: "three point one.three Limits of Visual-Only Feedback" #pg2.

Jane: So, they’re moving beyond simple testing environments and trying to simulate the actual user experience on hardware that has real limitations > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3.

Lu: This whole stack is designed so that research isn't siloed; it’s about creating a foundation where researchers can experiment with new decision-making mechanisms and see if they actually improve performance on real tasks > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Meng: If this works well at scale, it could mean we stop having to build completely separate agent systems for every single computer application > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1.

Conclusion: Tom: So to wrap up this OpenGCUA paper, we’re looking at a system that lets AI decide between visual observation and programmatic actions using code execution > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Jane: Exactly, it’s about giving the AI more freedom to choose its path on a computer by using code and visuals interchangeably > Page one of that work reads: "We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces" #pg1.

Lu: The implication here is that we’re moving toward a foundation where agents can handle complex computer tasks without being stuck just looking at screens > Page two of that work reads: "two Generalist CUA: Definition and Positioning" #pg2.

Meng: For us on the production side, this architecture gives us a blueprint to stop building separate systems for every different kind of user interface we have to support > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1.

Lalam: I’m pretty sure this is solid; the way it frames the decisions and how it tests on real macOS tasks shows high confidence in this direction > Page one of that work reads: "Model Capability Insights" #pg1.

Tom: The quantitative gains we saw, like that fifty-three percent reduction in steps per task on OSWorld-v2, are a big signal when you look at the numbers > Page one of that work reads: "Quantitative Gains" #pg1.

Jane: It really shows how much better these hybrid systems perform compared to just relying on visual observation alone > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3.

Lu: The openness of the stack is what makes it important; it’s not just one paper, but a whole way for the community to build and test these things > Page one of that work reads: "We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community" #pg1.

Meng: I just wonder how much effort it takes to maintain this open stack once it’s out there, but if it works, that’s what matters > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1.

Lalam: I think the main thing is the ability to combine those modalities; code execution isn't just an add-on, it’s central to how these agents operate now > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" #pg1.

Tom: So the core idea boils down to building this OpenGCUA stack for generalist computer use by making code execution and decision transfer central features > Page one of that work reads: "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use" #pg1.

Jane: It’s a really strong direction, showing that the future of agents isn't just about better models but about smarter tools that can actually use computers > Page three of that work reads: "The question has changed, from how well a model operates a GUI interface to how well a system completes work on a real computer by whatever means works" #pg3.

Lu: It sets the direction for what we need in agent design moving forward; it’s about that hybrid approach > Page two of that work reads: "two Generalist CUA: Definition and Positioning" #pg2.

Meng: So if you're building something, you can think about how to integrate code execution as a standard tool, not just an extra module > Page one of that work reads: "Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their ex" #pg1.

Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong

Alibaba Group

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-15

Code: https://github.com/ifuryst/open-codex-computer-use

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: As an excellent, fastidious, and diligent researcher, I have meticulously analyzed both provided excerpts concerning "OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use." My task is to

Key concepts

Generalist Computer-Use Agent (GCUA): This is the core agent architecture designed for general computer use. It allows the model to autonomously decide three things at every step: what to observe, how to act (choosing between GUI, CLI, or API), and how to organize its workflow. It aims for a unified system that handles diverse computer tasks flexibly.
Code Execution as a Key Feature: This capability lets the agent model directly apply its programming expertise to computer interaction. Instead of just describing actions, the model can write and execute code—combining GUI operations with shell commands or API calls—to perform complex, multi-faceted tasks in a single block.
Harness-to-Model Decision Transfer: This mechanism moves workflow control decisions from fixed harness rules directly into the model's decision-making process. The agent learns to decide when it needs to observe the environment (like requesting a screenshot) or how to track task state internally, giving it greater autonomy over its execution plan.

Terminology

Harness — 5× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes

Summary

As an excellent, fastidious, and diligent researcher, I have meticulously analyzed both provided excerpts concerning OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use. My task is to synthesize these pieces into a comprehensive, long, and detailed summary of the paper's contributions.

Here is the detailed synthesis:

Comprehensive Summary of OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

The research presented in OpenGCUA focuses on developing and open-sourcing an agent architecture designed for generalist computer use, aiming to unify the capabilities of large language models (LLMs) with direct, flexible interaction across both visual interfaces (GUIs) and programmatic environments. The core innovation is the Generalist Computer-Use Agent (GCUA), a system where the model autonomously decides how to observe its environment, how to act upon it, and how to organize its workflow across these diverse modalities.

Core Architectural Innovations:

Code Execution as a Key Feature: A central pillar of the OpenGCUA design is the integration of code execution. This capability allows the agent model not only to interact with graphical interfaces but also to write and execute programs that combine GUI operations with programmatic logic (e.g., using Python for GUI manipulation combined with shell commands). This moves beyond simple scripting, enabling models to directly apply their existing programming expertise to computer interaction.
Harness-to-Model Decision Transfer: The system incorporates a mechanism for harness-to-model decision transfer. This allows selected workflow control decisions—derived from fixed harness rules—to be moved into the model's decision-making process. This includes dynamic, on-demand observation (deciding when a new screenshot is necessary), task-state tracking via an internal todo interface, and action intent annotations to clearly explain the rationale behind code execution steps.
Generalist Agent Decision Making: The GCUA operates by making three critical decisions at every step: what to observe (choosing between a screenshot or an AX tree), how to act (selecting the appropriate interface: one program over GUI, AX tree, CLI, or API), and how to organize its work (planning the next steps, tracking progress, and managing context).

System Implementation and Evaluation:

The OpenGCUA harness is implemented using a combination of technologies: PyAutoGUI + bash within a Python runtime for general implementation, while the real-device harness leverages Playwright to facilitate robust browser interaction. Furthermore, a preliminary Cross-Device SDK has been developed to provide a unified programming interface across various platforms (computers, mobiles, browsers). A significant contribution is the development of a real-device harness, which addresses challenges inherent in real-world deployment—such as sharing machines with users and interacting with background windows—that are often absent in standard benchmark environments.

Key Findings and Performance Metrics:

The evaluation demonstrates substantial performance gains across various frontier models (including GPT-6 Astra and Claude Opus 5) when utilizing the full OpenGCUA configuration compared to a baseline GUI-only execution method.

Quantitative Gains: On the offline OSWorld-v2 set, the full GCUA setting yielded significant improvements: it raised the mean partial score by +19.6 points and reduced steps per task by an average of 53.0%.
Code Execution Efficacy: The integration of code execution proved valuable, adding 6.1 points to the mean partial score and cutting steps by 14.8% against a GUI+CLI baseline.
Model Capability Insights: Experimental results on four real macOS tasks revealed critical differences between models under the real-device harness regarding reasoning efficiency and task completion. Specifically, one finding highlighted a disparity in token usage: Qwen3.8-Max used 19.4 times as many reasoning tokens as GPT-6 Astra on four real macOS tasks, despite GPT-6 Astra completing the tasks in fewer steps (34.5 steps vs. 50.8 steps for Qwen3.8-Max). This suggests that the capability of code execution lies not just in writing executable code, but in its correct application to cover the entire task scope and verify completion, rather than merely possessing the ability to write loops and conditions.

Main Contributions:

The paper's main contributions are multifaceted:

Model Development: Creating a model architecture that unifies computer and browser use through code execution and harness-to-model decision transfer as its central design tenets.
System Development: Providing a functional, real-device harness that captures lessons learned from benchmark environments when transitioning to actual user machines.
Evaluation Trajectories: Releasing detailed evaluation trajectories from eight frontier models (e.g., GPT-6 Astra) to allow for in-depth analysis of code execution and decision transfer patterns.
Efficiency Studies: Systematically studying and improving GCUA execution efficiency through code execution, concise model reasoning, and harness optimization.
Real-World Evaluation: Developing evaluation environments on real devices (specifically macOS) to accurately assess agents' ability to perform tasks under authentic operating conditions.

In conclusion, OpenGCUA presents a significant step toward creating a truly generalist computer-use agent by empowering LLMs with the ability to dynamically choose between visual and programmatic interactions, underpinned by intelligent decision transfer mechanisms and rigorously tested on real-world hardware.

Improvements for AI systems

Bold header: Code Execution for Intermediate Results

The system can now apply their existing programming capabilities directly to computer interaction by using intermediate results to determine subsequent operations, as described in Section 3.1: intermediate results can decide the next operation without another model call.

Bold header: Harness-to-Model Decision Transfer

The agent gains autonomy by making workflow control decisions that were previously fixed rules; this is achieved through harness-to-model decision transfer, where the model decides when to request screenshots and how to organize task state instead of relying on a harness rule.

Bold header: Unified Cross-Interface Program Space

The system can execute complex, multi-faceted tasks by writing a program over all interfaces, meaning it can combine GUI actions, an accessibility operation, a shell command, and an API call within a single execution block.

Abstract

Computer use has reached a turning point: the focus is expanding from GUI-only interaction to systems that complete tasks on users' computers through visual and programmatic interfaces. We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces. Yet frontier implementations, such as GPT-6 Astra's computer-use system, remain largely closed, limiting systematic study of their design. We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community. This report presents the first release of OpenGCUA. Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their existing programming capabilities directly to computer interaction. We also propose harness-to-model decision transfer, moving selected observations and task-state decisions from fixed harness rules to the model. On OSWorld-v2, our full configuration improves task scores across all eight evaluated models, including GPT-6 Astra and Claude Opus 5, raising mean partial score by +19.6 points and reducing reported steps per task by an average of 53.0% relative to batched GUI-only execution. We release our implementation and prompts, evaluation trajectories from all eight models, and preliminary versions of a real-device harness and a cross-device SDK. The harness addresses real-device challenges not fully captured by benchmarks, while the SDK provides a common programming interface across computers, mobiles and browsers. Across four real macOS tasks under our real-device harness, selected Qwen3.8-Max runs use 19.4 times as many reported reasoning tokens as GPT-6 Astra, highlighting substantial differences between models in real-device execution efficiency.