SparsePixels: Efficient Convolution for Sparse Data on FPGAs

The gist The SparsePixels framework introduces a method for efficient convolution on FPGAs by selectively computing only on active input pixels, which reduces per-layer compute from O(HWK2CinCout) to O(n 2CinCout) and storage from O(HW C) to O(nC), achieving significant speedups for spatially sparse data in low-latency applications

Problem Statement

The core problem addressed is whether it is possible to improve the processing efficiency of CNNs for spatially sparse images on FPGAs to allow for accelerated real-time inference in experiments that require microsecond-level latency or below. This challenge arises because standard convolution requires scanning the entire input, which is inefficient when semantic information occupies only a small fraction of the pixels.

Contributions

The authors make three primary contributions to this area.

1. Development of a sparse convolution method designed for efficient FPGA implementation for applications in the broader particle physics experiments, targeting low inference latency that is independent of input data and is constant at runtime.

2. Proof-of-concept on spatially sparse image data, demonstrating significant FPGA inference speedups with comparable model performance.

3. A Python library for quantization-aware training of sparse CNNs and C++ HLS modules for FPGA deployment.

Sparse Convolution Method

The framework, SparsePixels, consists of five core sparse layers sufficient to build a sparse CNN. The process begins with the sparse input reduction layer, which dynamically retains up to N max active (a hyperparameter) active pixels into two sparse arrays: the feature array afeat (features) and the hash array ahash (height and width coordinates). This layer avoids data-dependent write addresses by using a recursive binary reduction scheme to read out active pixels, which results in a fixed-shape reduction tree. The sparse convolution layer performs a sparsity-preserving convolution that fixes the input-output sparsity, meaning an output pixel is active if and only if the input at the same pixel location is active, preventing dilation. The computation count scales as N max active 2CinCout rather than HWK2CinCout for standard convolution.

HLS Implementation Details

The HLS implementation focuses on storing only the active pixels in compact arrays and performing all subsequent computation on these arrays. The sparse input reduction layer uses a recursive binary reduction scheme to obtain the leftmost active pixel, which is then written into the first slots of afeat and ahash, with unused slots padded as invalid. The sparse convolution layer loops over output pixels and input pixels, computing offsets (∆h, ∆w) between their coordinates from the hash array to check if they lie within the K × K receptive field before performing a channel-wise dot product. This approach avoids explicit looping over all K squared kernel positions, making it advantageous for sparse data where active pixels can be far apart.

Evaluation Results

Experimental validation on a realistic neutrino dataset demonstrated substantial speedups. For example, the sparse-large model achieved a ×73 speedup over the standard CNN with comparable or slightly lower accuracy. The latencies and initiation intervals (II) of the sparse CNNs are frozen at synthesis, meaning they are data-independent and constant at runtime. This constant runtime property is crucial for trigger systems in modern particle experiments that enforce strict latency bounds. The framework is designed to benefit future algorithm development for efficient data readout in modern experiments with strict latency requirements of microseconds or below.

Limitations and Future Work

Current limitations include the full parallelization in HLS, which limits input and model sizes, as the current implementation prioritizes minimal latency at the cost of higher resource usage. A key area for improvement is enabling flexible control over the degree of parallelization in every sparse layer so one can trade between latency and resource utilization. Furthermore, choosing a proper N max active is currently empirical; a possible improvement is to make N max active, and potentially the threshold, trainable hyperparameters in the training library. Coarse quantization is also noted as an area for future upgrade to adopt HGQ for finer-grained per-weight quantization.

Summary

SparsePixels introduces a framework for sub-microsecond FPGA inference of CNNs using sparse convolution on spatially sparse 2D data. The key features in the HLS implementation are (1) dynamically identifying and retaining a small subset of active pixels from a large input into compact arrays, then (2) performing all subsequent computation only on those retained pixels. By selectively computing on the informative subset of pixels, the design avoids computation on empty regions, yielding significant inference speedups and resource savings. Our design operates on a fixed pixel budget that does not change between inputs, so the hardware runtime is input-independent and constant by construction. On a representative example of MicroBooNE neutrino images with 4k input pixels, a sparse CNN that computes on at most 20 active pixels achieves a ×73 inference speedup on an FPGA over a standard CNN with the same base architecture (133 vs. 9733 clock cycles; 0.665 vs. 48.665 µs), with < 2% ROC AUC drop, and resource utilization well within device budgets. For easy adoption, we provide a Python library supporting quantization-aware training of sparse CNNs and HLS C++ modules for FPGA deployment. We hope this work benefits experiments operating under stringent sub-microsecond latency constraints, such as modern particle experiments, where standard CNNs are often infeasible due to their high latency, which would otherwise force substantial performance compromises.

1. Sparse Convolution for Neutrino Event Identification: The SparsePixels framework enables real-time inference on FPGAs for neutrino identification, achieving a speedup of a factor of 73 (0.665 µs vs 48.665 µs) while maintaining comparable accuracy, which is crucial for ultra-low latency trigger applications.

2. Constant Latency Inference: By operating on a fixed pixel budget that is frozen at synthesis, the inference latency becomes data-independent and constant at runtime, solving the problem of input-dependent computation that plagues existing methods in strict microsecond latency domains.

3. Hardware Acceleration for Sparse Data Readout: The sparse input reduction layer uses a recursive binary reduction scheme to read out active pixels, replacing data-dependent write addresses with a fixed-shape reduction tree, ensuring hardware efficiency and predictable performance regardless of the input sparsity level.

4. Efficient Model Deployment via HLS: The framework provides a Python library for quantization-aware training of sparse CNNs and C++ HLS modules for FPGA deployment, allowing researchers to easily translate models into high-level synthesis firmware while supporting fixed precision deployments.

Inference of standard convolutional neural networks (CNNs) on FPGAs often incurs high latency and a long initiation interval due to the deep nested loops required to densely convolve every input pixel regardless of its feature value. However, input features can be spatially sparse in some image data, where semantic information may occupy only a small fraction of the pixels and most computation would be wasted on empty regions. In this work, we introduce SparsePixels, a framework that implements sparse convolution on FPGAs by selectively retaining and computing on a small subset of active input pixels while ignoring the rest, reducing the per-layer compute from O(HWK 2C inC out) to O(n 2C inC out) and the storage from O(HWC) to O(nC) when n squared HW. Because computation always runs over a single pre-specified pixel budget that is frozen at synthesis, the inference latency is data-independent and constant at runtime. We show that, for identifying neutrino interactions in naturally sparse LArTPC images with 4k pixels, a standard CNN with a compact size of 4k parameters incurs an inference latency of 48.665 μ s on an FPGA, whereas a sparse CNN of the same base architecture, computing on less than 1% of the input pixels, achieves a times 73 speedup to 0.665 μ s with resource utilization well within on-chip budgets, trading only a small percent-level performance loss. This work aims to benefit future algorithm development for efficient data readout in modern experiments with strict latency requirements of microseconds or below.