ANEForge: Python for direct computation on the Apple Neural Engine

The gist The ANEForge package provides Python for direct computation on the Apple Neural Engine, enabling users to program it directly and without CoreML, which bypasses the sanctioned runtime scheduling option.

Motivation and significance

The Apple Neural Engine is a fixed-function neural accelerator that ships on every recent iPhone, iPad, and Apple Silicon Mac [1]. The only sanctioned route to it is CoreML, which treats it as a scheduling option rather than a target [3]. ANEForge compiles a lazy tensor graph into a single ANE program and dispatches it through the same ANE daemon and kernel-driver stack as Apple’s internal framework, with no CoreML dependency [2]. This allows for direct access to the engine via private symbols, which is not exposed to application code [1].

Software description

The software architecture revolves around a single path: a lazy operator graph is lowered to a single ANE program and dispatched to the hardware [2]. The process involves constructing a graph of Tensor nodes from the operator surface, lowering it to the Model Intermediate Language (MIL) that the ANE compiler consumes, packing weights into one binary blob, and producing a callable program handle [2]. Dispatch utilizes a lower-latency C interface, e5rt, exported by Apple’s private Espresso framework [2]. This path allows for fixed dispatch units rather than heuristic selection, enabling characterization of the engine's speed and power [2].

Software functionalities

ANEForge offers several key capabilities validated on Apple Silicon against a NumPy or framework reference [3]. These functionalities include:

1. Graph compilation and dispatch: af.compile lowers a graph to one ANE program and returns a callable, while af.input and af.image input declare input ports that emit an on-engine dequantization for raw camera or decoded-video bytes [2].

2. Weight compression: The package streams int8, int4 lookup-table, or unstructured-sparse weights from the engine’s own dequantization path rather than folding them to half precision at compile time [3]. This compression can reduce the weight footprint by about 4 times for int4 and 2 for int8 [3].

3. Native attention: af.sdpa targets the engine’s dedicated fused-attention layer, for causal attention, the single-query decode shape over a cached key and value, and a runtime additive mask [2]. This reaches the native fused layer that the public toolchain decomposes into matmul, softmax, matmul [3].

4. Resident on-engine state: Two kinds of state stay resident on the engine across steps through output-to-input buffer aliasing: a zero-copy key-value cache for autoregressive decoding, and optimizer state for training [2].

5. On-engine training: The autograd module runs the forward pass, the backward pass, and the optimizer update on the engine [2], utilizing a fused softmax-cross-entropy with an analytic, half-precision-stable gradient [3].

Illustrative examples

The package supports a build-then-compile-then-call pattern demonstrated in Listing 1 of Figure 3, where operations like af.input((1, 3, 32, 32)) conv relu mean(2, 3) are used to build a lazy graph [4]. Calling net(image) dispatches the compiled program to the engine [4]. Pretrained models such as ResNet-18 and Stable Diffusion U-Net run end-to-end against framework references, with validation showing high cosine similarity against float32 outputs [5]. For instance, a pretrained ResNet-18 forward runs in 0.33 ms on the engine, drawing 4.5W on the ANE rail during the hot loop [5].

Impact

ANEForge provides capabilities that the sanctioned route does not, such as a dispatch unit fixed programmatically [6], and native layer types the public compiler never emits [6]. It supports two kinds of study: a machine-checked census of which operators the engine accepts, built on the package’s conformance gate, and a characterization of the engine’s speed and energy across workload regimes [6]. Furthermore, on-engine training keeps the data, the gradients, and the optimizer state on the device, which is a substrate for research on private personalization using the lowest-power programmable compute block of the system-on-chip [6]. The package targets research and local development since software linking private frameworks cannot ship through the App Store [6].

Limitations

The package calls private, undocumented Apple symbols that carry no API contract and can change between operating-system releases [5]. A release is verified against a recorded macOS and ANE-compiler version, with the operator set as the regression gate [6]. The compute path is half precision, meaning half-precision sets the dataplane’s numerical reach, although accumulation error stays acceptable for validated models [7]. The engine exposes a single hardware lane, so throughput gains come from fusing work into one program rather than concurrent dispatch [7]. The operator surface is the set the hardware accepts under the conformance gate, not a general tensor library, meaning a graph outside that set fails at compile time [7].

Conclusions

ANEForge gives Python direct, CoreML-free access to the Apple Neural Engine: a lazy operator graph compiles to a single ANE program and dispatches through the private framework stack that CoreML uses internally, with the compute unit fixed [8]. The package supports inference, on-device training, and dense numerical linear algebra and spectral methods, each validated against a reference [8]. Call overhead sits near the engine’s dispatch floor, and ResNet-18, a sentence encoder, a Vision Transformer, and the forward pass of a Stable Diffusion U-Net run end-to-end on the hardware [8]. The private symbols carry no API contract, so each release is verified against a recorded macOS and ANE-compiler version, with the operator set as the regression gate [8].

Declaration of competing interest The author declares no competing financial interests or personal relationships that could have influenced the work reported in this paper.

1. textbf Direct ANE Program Dispatch without CoreML Dependency: The system can execute workloads directly on Apple Silicon hardware without relying on CoreML for scheduling, as ANEForge compiles a lazy tensor graph, built from 58 fused operators and 19 native bridge operators, into a single ANE program and dispatches it through the private stack.

2. textbf Native Fused Attention Implementation: The system gains access to the ANE's native fused attention, which allows for optimized causal attention using af.sdpa targets the engine’s dedicated fused-attention layer, achieving up to 3.89 times faster performance on Transformer attention blocks compared to unoptimized graph lowerings.

3. textbf On-Engine Training and State Management: The system supports full training loops by running the forward pass, the backward pass, and the optimizer update of training on the engine, keeping decoder and optimizer state resident across steps via output-to-input buffer aliasing.

4. textbf Weight Compression for Inference: Inference can be accelerated with specific weight formats by using methods like af.compile(y, compress=int8") which streams weights from the engine’s dequantization path, resulting in a weight footprint reduction of about 4 times for int4 and 2 for int8."

5. textbf Fixed-Iteration Numerical Kernels: The system can execute dense linear algebra and spectral methods using fixed-iteration Krylov solvers and a factored fast Fourier transform as static-dataflow graphs, ensuring these operations run at a program size independent of the problem dimension.

ANEForge is a Python package that programs the Apple Neural Engine (ANE), the fixed-function neural accelerator on every recent Apple device, directly and without CoreML. In production the engine is reachable only through CoreML, which treats it as a scheduling option: no configuration requires the ANE, and a model can silently run on the CPU or GPU instead. ANEForge compiles a lazy tensor graph, built from 58 fused operators and 19 native bridge operators, into a single ANE program. The program is dispatched through the same ANE daemon and kernel-driver stack as Apple's internal framework. Beyond inference, the package reaches the engine's native fused attention, streams int8, int4, and sparse weights, keeps decoder and optimizer state resident across steps, and runs the forward pass, backward pass, and optimizer update of training on the engine. A small fused program completes a call in about 90us, near the engine's 70us per-program dispatch floor, and a pretrained ResNet-18 forward runs end-to-end in 0.33ms. ResNet-18, a sentence encoder, and a Vision Transformer run end-to-end against framework references, and a Stable Diffusion U-Net validates its forward pass. ANEForge targets Apple Silicon under macOS 14 and later. Each release is verified against a recorded macOS and ANE-compiler version.