Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices

The gist

Minions presents a dynamic and distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads, enabling teams to leverage existing hardware without requiring additional GPU procurement.

System Overview and Architecture

Minions consists of four primary components: Minion Server, PostgreSQL Metadata Store, Minion Executor, and Minion Watchdog The system architecture consists of four primary components: Minion Server, PostgreSQL Metadata Store, Minion Executor, and Minion Watchdog Users or IR pipelines submit GPU inference tasks through the Minion Server The system workflow illustrates the end-to-end process of GPU inference task submission, execution, and failure recovery.

Key Components

The Minion Server serves as the central coordination layer of the system, handling all user-facing requests through a web UI and APIs Upon task submission, the Minion Server asynchronously splits a task into subtasks and enqueues them into the PostgreSQL metadata store. The Minion Executor is the primary GPU execution agent and runs on macOS devices equipped with Apple Silicon GPUs Executors periodically fetch eligible subtasks via REST APIs exposed by the Minion Server For embedding generation, Minions leverages open-source transformer models from HuggingFace. For large language model (LLM) completion workloads, Minions integrates with Ollama, an opensource framework for running LLMs locally on macOS devices. The Minion Watchdog is responsible for monitoring active subtask executions and handling executor failures.

Data Storage and Execution Model

Minions utilizes AWS S3 as the primary storage layer for GPU inference inputs and outputs Task input artifacts are read from S3 by executors, and inference results are written back to S3 upon completion. This design decouples compute from storage, avoids transferring large payloads through the Minion Server, and leverages industrystandard access control and security mechanisms. The Minion Executor uses a pull-based execution model which accommodates the inherently intermittent availability of personal devices while enabling elastic parallelism.

Subtask Acquisition and Task Binding

Minions employs an atomic subtask acquisition mechanism implemented with a dedicated table, minion subtask acquirement, that records which executor currently owns a given subtask. This approach provides a simple, low-overhead mutual exclusion mechanism using standard database constraints. Dynamic task binding allows executors to express preferences for a specific team, project, and task name These preferences are evaluated by the Minion Server as part of subtask selection logic and translated into database-level filters. By default, Minions processes inference workloads in a first-in, first-out (FIFO) order.

Observability and Operational Impact

Minions incorporates observability through Grafana-based dashboards that provide real-time visibility into system state at a team level. All observability metrics are derived from the PostgreSQL Metadata Store, which serves as the system’s source of truth. Executor-level metrics include the number of registered, online, busy, and idle executors. The operational impact shows that with Minions, GPU queue time can be reduced significantly compared to before adoption. This demonstrates that idle Apple Silicon GPUs on macOS devices can be aggregated into a dynamic, distributed virtual GPU cloud for machine learning inference.

Future work will extend Minions to support distributed model training in addition to inference. This platform points toward a novel opportunity for the machine learning community if idle macOS devices across organizations can be securely federated, they could form a massive, shared GPU resource pool that benefits the broader ecosystem. The paper concludes that Minions demonstrates that idle Apple Silicon GPUs on macOS devices can be aggregated into a dynamic, distributed virtual GPU cloud for machine learning inference.

Acknowledgments

We thank Scott Guo for peer review and helpful comments, and Matt Jockers for peer review and publication coordination.

References

[1] Hugging Face: https://huggingface.co/

[2] Ollama: https://ollama.com/

[3] PostgreSQL: https://www.postgresql.org/

[4] Grafana: https://grafana.com/

--- Page 1 ---

Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices

Jianhua Li, Yun Xing, Girisha De Silva, Ryan Flint, Kai Yuan, Sean Suchter

Apple

USA

jianhua li3@apple.com

Apple USA jianhua li3@apple.com Yun Xing Apple USA yun xing@apple.com Girisha De Silva Apple USA girisha desilva@apple.com Ryan Flint Apple USA rflint@apple.com Kai Yuan Apple USA kai yuan@apple.com Sean Suchter Apple USA ssuchter@apple.com Abstract The increasing demand for GPU-accelerated machine learning inference has exposed limitations in centralized GPU infrastructures, including high cost, limited elasticity, and long wait times under resource constraints At the same time modern Apple Silicon–based macOS devices are equipped with powerful GPUs that remain largely idle during normal usage This paper presents Minions, a dynamic and distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads Minions enables teams to leverage existing hardware without requiring additional GPU procurement The system provides flexible GPU inference task execution, fault tolerance, and efficient resource utilization Our experience shows that idle macOS devices can be transformed into a practical distributed GPU inference platform in environments where centralized GPU resources are constrained Keywords GPU inference, distributed systems, Apple Silicon, information retrieval systems, large language models

--- Page 2 ---

Jianhua Li, Yun Xing, Girisha De Silva, Ryan Flint, Kai Yuan and Sean Suchter

Figure 1: Minions architecture overview.

1. Apple Silicon GPU utilization for large language model (LLM) inference will be significantly increased by dynamically aggregating idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads. This enables teams to leverage existing hardware without requiring additional GPU procurement.

2. Implementation of a dynamic task binding mechanism allows executors to express preferences for specific workloads, enabling them to preferentially dequeue and execute matching subtasks, even if other tasks are ahead in the global queue. This optimizes resource allocation based on team or project priorities.

3. The system supports heterogeneous inference types by leveraging both open-source transformer models via the HuggingFace API for embedding generation and integrating Ollama for large language model (LLM) completion workloads. This allows Minions to handle both embedding generation and large language model (LLM) completion tasks locally on the same distributed platform.

4. The architecture incorporates robust fault tolerance through a Minion Watchdog that reclaims unfinished subtasks upon executor failure, ensuring reliable task completion in the presence of ad hoc device churn. This minimizes lost work due to intermittent availability of personal devices.

5. Decoupling compute from storage by using AWS S3 as the primary storage layer for inputs and outputs simplifies data management while supporting scalable and secure execution. This design ensures that large payloads are handled efficiently without transferring them through the Minion Server.