FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving

arXiv:2610.22143 · cs.CV · Submitted 2026-10-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving".

Jane: The gist: A perception system that must brake for a pedestrian does not need a low average latency, but must deliver an answer before the deadline on every frame,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up, the paper "FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving" is all about making latency deterministic on edge hardware by explicitly bounding the compute in programmable logic and using a watchdog to enforce delivery deadlines.

Jane: The core implication is that you don't need low average latency when you have a hard deadline, especially when dealing with systems where missing that deadline is dangerous, like braking for a pedestrian one <ref:2610.22143#pg2>.

Lu: They achieved this by decomposing the unpredictability into architecture and platform sharing, and they implemented fixed-trip-count IPs for operations the DPU couldn't handle, which they numerically matched to their software references two <ref:2610.22143#pg2>.

Meng: For those of us shipping models, it shows that locking down parts of the inference graph into static operations can give you a much better guarantee than just relying on dynamic scheduling or hoping for low OS jitter two <ref:2610.22143#pg2>.

Lalam: I'm confident in the result because they show that this deterministic approach, even at a certain compute cost, gives you certifiable timing bounds over one hundred thousand frames one.

Jane: It’s about moving from just measuring performance to proving it meets a specific time constraint under worst-case conditions one <ref:2610.22143#pg2>. That's what makes this paper important for real world deployment.

Conclusion: Tom: So, this paper is about FALCON, which is an FPGA front-end for occupancy networks on camera data one.

Jane: It sounds complicated because of all that hardware talk, but really it's about making sure a perception system doesn't miss a pedestrian when it needs to brake one.

Tom: Exactly. And the authors are focusing on latency guarantees, not just speed. They’re using this AMD Kria KR260 board and turning programmable logic into something cycle-bounded one.

Jane: That’s the big shift from just trying to get a low average latency to actually proving an answer will arrive before a deadline every single time, even in the worst cases one.

Tom: Right. They use this layered approach—analytical bounding for the programmable logic, measured envelopes for inference, and then a watchdog that flags stale data if things run late one.

Jane: It means they’re tackling the problem of nondeterminism by explicitly defining where it happens, whether it's the hardware architecture or how the software runs on top of it one.

Tom: And they do this by adding two fixed-trip-count programmable logic IPs that handle things the main DPU can't manage, like view transforms and resizing one.

Jane: That’s a tangible thing for an engineer to look at—taking those hard software references and making them fixed hardware paths one.

Tom: The results are pretty strong. They got a mIoU of twenty-nine point eighty-eight which is ninety-two point eight percent of what the DPU couldn't even do on paper one.

Jane: And they showed that this setup stays within fifteen times its median performance across different hardware compared to other systems like Jetson Orin or datacenter GPUs one.

Tom: It’s a lot of data showing how you can enforce these hard time constraints on commodity edge accelerators one.

Jane: It really changes the conversation from just "how fast is it?" to "when will I get my answer?" and that's a huge difference for safety systems.

Tom: If we can make this kind of latency guarantee practical on the edge, it opens up real deployment possibilities for critical autonomous features one.

Haohua Que, Mingkai Liu, Handong Yao

College of Engineering, University of Georgia

cs.CV

Submitted: 2026-10-15

Updated: 2026-10-16

Comments: Accepted to WACV 2027 (Applications track). 10 pages

Code: https://github.com/Que121/falcon-kr260-release

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist: A perception system that must brake for a pedestrian does not need a low average latency, but must deliver an answer before the deadline on every frame, the worst case

Key concepts

Latency Determinability: This is the ability to guarantee that a computation will finish within a specific time limit, regardless of input changes or system noise. FALCON achieves this by bounding the programmable logic (PL) compute analytically and measuring inference latency over extensive traces, moving beyond just optimizing average performance.
Layered Latency Guarantees: The system enforces timing constraints through multiple mechanisms. Programmable Logic (PL) is bounded mathematically for cycle-level certainty. Inference latency is verified as a measured envelope across 100k frames. Finally, a watchdog ensures output delivery, flagging stale data if the deadline is missed.
Fixed-Trip-Count PL IPs: The system implements two specific operations—view-transform gather and bilinear resize—as fixed hardware IPs. These are numerically matched to their software references and have a known, fixed number of computational trips. This eliminates nondeterminism in these critical steps that commodity accelerators often struggle with.
Worst-Case Deadline Enforcement: Instead of aiming for low average latency, FALCON focuses on meeting the deadline for every single frame, including the worst-case scenario. It uses a fixed-work proxy of six ResNet-50 inferences per frame to ensure that across three 100k-frame runs, all 300k proxy frames delivered an output within the strict 100 ms deadline plus a small buffer.

Summary

The gist: A perception system that must brake for a pedestrian does not need a low average latency, but must deliver an answer before the deadline on every frame, the worst case included<ref:2610.22143#pg2>.

How it works

FALCON builds a camera-based 3D occupancy front-end on the AMD Kria KR260 embedded FPGA to make latency guarantees layered and explicit<ref:2610.22143#pg2>. The system addresses the limitation of commodity edge accelerators which optimize the mean through dynamic scheduling, OS jitter, thermal throttling, and contention, meaning their per-frame latency can be measured but not bounded<ref:2610.22143#pg2>. The latency guarantee is layered: programmable-logic (PL) compute is cycle-bounded analytically; inference is a measured envelope over 100kframe traces; and a watchdog enforces output delivery, publishing the previous output flagged stale when a frame is late<ref:2610.22143#pg2>.

Key Contributions

The paper makes three main contributions to the field<ref:2610.22143#pg3>.

**: An evaluation axis for safety-critical occupancy<ref:2610.22143#pg6>. This contribution focuses on Latency-determinability, which is driven by input-dependent computation, not sparsity<ref:2610.22143#pg6>. Nondeterminism is decomposed into architecture, platform sharing, and host path<ref:2610.22143#pg6>. The paper also introduces two fixed-trip-count PL IPs that close the DPU gaps<ref:2610.22143#pg6>. These are the Lift-splat view transform and BEV-neck bilinear resize, implemented as custom HLS IPs with cycle-bounded, input-invariant compute numerically matched to their software references<ref:2610.22143#pg6>. Finally, it presents a quantized board deployment and a measured, enforced deadline<ref:2610.22143#pg6>. Every stage is deployed and measured on the board, and the deadline behavior is characterized at 100k-frame scale against GPU and Orin<ref:2610.22143#pg6>. The full pipeline composes to about 1.3 s per frame on the board<ref:2610.22143#pg6>. The board reaches mIoU 29.88, which is 92.8% of an FP32 baseline that the DPU cannot run<ref:2610.22143#pg6>. The KR260 tail stays within 1.5× its median at any window, while Jetson Orin reaches 4.4× and the worst datacenter GPU run reaches 23.6×<ref:2610.22143#pg2>. The two operations the DPU cannot run, a view-transform gather and a bilinear resize, become fixed-trip-count PL IPs numerically matched to their software references<ref:2610.22143#pg6>. The accuracy survives a DPU-compatible INT8 recipe with calibration-time activation clipping and no QAT<ref:2610.22143#pg6>. Only PL compute is bounded analytically, while inference latency is a measured envelope<ref:2610.22143#pg6>. Delivery is enforced by a watchdog that publishes the previous output flagged stale when a frame is late<ref:2610.22143#pg6>. The deadline behavior at 100k-frame scale shows the raw tail crosses the budget on 0.088% of frames, with delivered-output curve terminating at 107.8 ms<ref:2610.22143#pg8>. The full pipeline thus composes to about 1.3 s per frame<ref:2610.22143#pg8>. The board exceeds the idealized reference at every evaluated scale, with pedestrian and motorcycle IoU exceeding it too<ref:2610.22143#pg8>. The fallback is priced directly, as ego-warping the previous on-board prediction by one and two 2 Hz keyframes produces a degradation of about 0.003 percentage points in expectation<ref:2610.22143#pg8>. The measured power from INA260 board telemetry is 4.33 W idle, 5.8 to 6.6 W under inference, and 8.7 W peak<ref:2610.22143#pg8>. The shared A100 runs FlashOcc at 33.5 ms median, meeting the 100 ms budget on that trace and missing 0.5% at 50 ms<ref:2610.22143#pg8>. The full pipeline thus composes to about 1.3 s per frame<ref:2610.22143#pg8>. The input downscaling does not buy this back, as halving the trunk input without retraining collapses accuracy<ref:2610.22143#pg8>. A 10 Hz tier therefore requires a retrained reduced-grid model, a measured engineering boundary rather than a knob<ref:2610.22143#pg8>. The HLS sources, deployment recipe, and raw 100k-frame traces are released at github.com/Que121/falcon-kr260-release<ref:2610.22143#pg9>. The timing-closed K26 bitstream co-places the DPU and both resize IP instances, so the full pipeline has not run as one bitstream<ref:2610.22143#pg9>. The evidence is dataset replay on hardware, not a moving-vehicle test<ref:2610.22143#pg9>. The paper claims no recall level is sufficient since it comes from the system safety case<ref:2610.22143#pg6>. Determinism does not remove compute cost, but fixed rigs and tensor shapes turn LSS splatting and known-shape resampling into static-index gathers with certifiable trip counts<ref:2610.22143#pg9>. The HLS sources, deployment recipe, and raw 100k-frame traces are released at github.com/Que121/falcon-kr260-release<ref:2610.22143#pg9>. The timing layer is model-independent, so a more accurate fixed-work model inherits it unchanged<ref:2610.22143#pg6>. The platform comparison is not matched in precision or isolation, as GPUs run FP16 and datacenter runs share cluster nodes<ref:2610.22143#pg6>. A real-time-configured GPU, Orin’s DLA, or another FPGA would be fairer baselines<ref:2610.22143#pg6>. The paper claims no recall level as sufficient since it comes from the system safety case<ref:2610.22143#pg6>. Determinism does not remove compute cost, but fixed rigs and tensor shapes turn LSS splatting and known-shape resampling into static-index gathers with certifiable trip counts<ref:2610.22143#pg9>. The HLS sources, deployment recipe, and raw 100k-frame traces are released at github.com/Que121/falcon-kr260-release<ref:2610.22143#pg9>. The timing layer is model-independent, so a more accurate fixed-work model inherits it unchanged<ref:2610.22143#pg6>. The platform comparison is not matched in precision or isolation, as GPUs run FP16 and datacenter runs share cluster nodes<ref:2610.22143#pg6>. A real-time-configured GPU, Orin’s DLA, or another FPGA would be fairer baselines<ref:2610.22143#pg6>. The paper claims no recall level as sufficient since it comes from the system safety case<ref:2610.22143#pg6>. Determinism does not remove compute cost, but fixed rigs and tensor shapes turn LSS splatting and known-shape resampling into static-index gathers with certifiable trip counts<ref:2610.22143#pg9>. The HLS sources, deployment recipe, and raw 100k-frame traces are released at github.com/Que121/falcon-kr260-release<ref:2610.22143#pg9>.

Improvements for AI systems

A perception system that must brake for a pedestrian can achieve a latency guarantee by layering its bounds: programmable-logic (PL) compute is cycle-bounded analytically; inference is a measured envelope over 100kframe traces; and a watchdog enforces output delivery. This allows the system to meet hard deadlines even when dealing with unpredictable hardware jitter or OS contention.
Implement two fixed-trip-count PL IPs for operations the DPU cannot run, specifically Lift-splat view transform [16, 34] and BEV-neck bilinear resize, which are custom HLS IPs with cycle-bounded, input-invariant compute. This moves critical geometric transformations off the jittery host path onto deterministic hardware.
Utilize a measured deadline enforcement mechanism: a watchdog enforces output delivery, publishing the previous output flagged stale when a frame is late. This ensures that even if the system misses its primary deadline, it always publishes a result within an enforcement envelope of 0.3 to 1.9 ms typical, 7.6 to 10.2 ms worst across the 421 events of three enforced 100k-frame runs.
Decompose nondeterminism into architecture and platform sharing: Address that input-dependent computation, not sparsity, drives tails and that nondeterminism is decomposed into architecture, platform sharing, and host path, leading to a more predictable system where the latency tail is controlled by fixed work rather than scene content.
Deploy a quantized model using no QAT [2, 3, 53] by employing calibration-time activation clipping instead of training-heavy methods. This keeps the deployment aligned with the DPU constraints and ensures accuracy preservation in an environment where per-tensor power-of-two scales and min-max calibration only are supported.