Towards bio-inspired visual navigation: a neuromorphic navigation paradigm for MAVs

The gist The proposed method presents the first end-to-end event-driven neuromorphic paradigm for MAV visual navigation in complex environments by integrating neuromorphic vision, spiking neural networks, and neuromorphic processors.

Motivation

Biological visual navigation achieves remarkable robustness at extremely low power, whereas frame-based robotic systems relying on artificial neural networks remain limited in both respects, particularly for energy-constrained micro aerial vehicles (MAVs) operating in complex environments<ref:2610.21978#pg6>. Current mainstream navigation systems based on conventional frame-based vision and artificial neural networks acquire image frames at fixed intervals and process visual information through dense and synchronous computation, as shown in Fig. 1B<ref:2610.21978#pg7>. This frame-driven paradigm makes it difficult to simultaneously achieve robust perception and high energy efficiency on high-speed, resource-constrained MAVs<ref:2610.21978#pg8>. The mismatch in sensing and computation principles may not only constrain navigation performance in increasingly challenging environments, but also limit the potential of conventional architectures to evolve toward more adaptive, efficient, and biologically inspired autonomy<ref:2610.21978#pg4>.

Proposed Paradigm

The proposed method first extracts the environmental representation from event streams using a spiking perceptual representation module<ref:2610.21978#pg5>. This representation is fed into a spiking navigation module, where guidance learning and population coding enable efficient policy learning with few time steps<ref:2610.21978#pg6>. The navigation module then outputs a waypoint converted into executable trajectory points<ref:2610.21978#pg6>. The system integrates an event camera, a spiking neural network, and a neuromorphic processor into a closed-loop navigation system to mimic biological visual sensing and computation<ref:2610.21978#pg6>.

Key Contributions

The main contributions of this paper are as follows:

1. First, we present the first end-to-end event-driven neuromorphic visual navigation paradigm for MAVs that has been successfully transferred from simulation to real-world deployment<ref:2610.21978#pg6>.

2. Second, we develop an onboard spiking perceptual representation module that converts asynchronous event streams into compact geometric features for MAV navigation by compensating for the absence of explicit environmental geometric structure in raw event streams<ref:2610.21978#pg6>.

3. Third, we propose an end-to-end MAV navigation policy based on spiking neural networks, enabled by efficient guidance learning and population coding<ref:2610.21978#pg6>.

System Architecture and Training

The physical implementation of the end-to-end event-driven neuromorphic navigation system consists of an event camera (Prophesee EVK4), a neuromorphic processor (Intel Kapoho Bay), an edge computing platform (NVIDIA Jetson Orin NX), an x86-based host computer, and a micro aerial vehicle<ref:2610.21978#pg7>. The system is partitioned into two modules: the perceptual representation module for depth estimation and feature extraction deployed on the NVIDIA Jetson Orin NX in the form of simulated SNNs, and the navigation module deployed on the Intel Kapoho Bay neuromorphic processor<ref:2610.21978#pg8>. The training pipeline involves transforming grayscale image sequences into asynchronous event streams using an event simulator<ref:2610.21978#pg9>. Depth estimation is introduced as an auxiliary task to enable geometry-aware spiking features learning from event streams<ref:2610.21978#pg9>. The learned event features are fused with the MAV state and goal direction, and then encoded into spike trains through a multi-Gaussian population coding scheme with trainable mean and bias<ref:2610.21978#pg10>.

Navigation Policy Optimization

The spiking navigation network is trained with two supervision objectives: the geometric loss Lgeo and the trajectory loss Ltra<ref:2610.21978#pg6>. The geometric loss enforces consistency between the network-predicted depth and the ground-truth depth, improving scene representation<ref:2610.21978#pg6>. The trajectory loss Ltra consists of three terms: the smoothness loss Ls, the safety loss Lo, and the goal loss Lg<ref:2610.21978#pg6>. Guidance learning is employed to provide physically meaningful gradients for navigation policy optimization<ref:2610.21978#pg6>. The smoothness loss is defined as Ls = A T H A, where H denotes the smoothness cost matrix in polynomial coefficient space<ref:2610.21978#pg6>. Obstacle avoidance is handled by a distance penalty function fdp (p(t)) = e(-d(t)-d0/κ), and the safety loss Lo is defined as Lo = ∑ k fdp (p(kΔt))Δt<ref:2610.21978#pg6>. The goal loss is defined as Lg = (end state, p − pgoal) squared<ref:2610.21978#pg6>. Since the spike-generation function is non-differentiable, a surrogate gradient method is adopted to enable backpropagation through the spiking neural network<ref:2610.21978#pg6>.

Performance and Efficiency

Experiments show that the proposed neuromorphic method achieves navigation performance comparable to ANN-based methods while markedly reducing power consumption<ref:2610.21978#pg6>. The spiking perceptual module consumes only 19.57% of the power of its ANN counterpart, while the spiking navigation module deployed on Intel Loihi consumes only 12.37% of that required by its ANN counterpart running on NVIDIA Orin NX<ref:2610.21978#pg6>. For the perceptual representation network, the SNN-based network requires only 0.507 mJ per inference, which is substantially lower than the ANN-based model consuming 2.591 mJ per inference<ref:2610.21978#pg9>. Similarly, for the navigation network deployed on conventional platforms, the per-inference energy consumption of the ANN-based model is approximately 8.09 times that of the SNN-based model in high-performance mode (25 W)<ref:2610.21978#pg9>.

Real-World Validation

Indoor and outdoor navigation tests were conducted by placing the target point behind a series of obstacles, requiring the MAV to avoid the obstacles and successfully reach the target<ref:2610.21978#pg6>. The proposed method successfully reached the target in seven out of ten indoor trials and nine out of ten outdoor trials<ref:2610.21978#pg6>. Compared with simulation experiments, the navigation success rate decreased in both real-world settings, mainly due to two factors: limited horizontal field of view from the Prophesee EVK4 lens and less reliable depth maps captured by the Intel RealSense D435i<ref:2610.21978#pg7>. The work demonstrates that the proposed neuromorphic navigation paradigm can be successfully transferred from simulation to physical flight<ref:2610.21978#pg6>.

Limitations and Future Work

Despite these results, the current physical system still has two main limitations: first, full onboard deployment of all spiking neural networks on neuromorphic hardware has not yet been achieved due to limited neuron models and capacity, necessitating additional wireless data transmission<ref:2610.21978#pg6>. Second, pose estimation still depends on a conventional frame-based camera because existing monocular event-based visual odometry remains limited in estimation accuracy and computational cost<ref:2610.21978#pg6>. Future work will explore more capable neuromorphic hardware with greater capacity to enable more complete onboard neuromorphic deployment, while also developing lightweight and accurate event-based visual odometry to extend event-based perception to state estimation<ref:2610.21978#pg6>. The work moves neuromorphic MAV navigation beyond isolated sensing or computation modules toward a unified perception–decision–execution loop<ref:2610.21978#pg6>.

Materials and Methods

This section presents the complete pipeline of the end-to-end event-driven neuromorphic navigation method, including event cameras, spiking neural networks, population coding for spike trains encoding and decoding, state and action spaces, and the end-to-end training procedure<ref:2610.21978#pg6>. The detailed procedure for power consumption measurement includes operation-count-based energy estimation and hardware-based dynamic power measurement<ref:2610.21978#pg6>. The hardware deployment uses Ubuntu 20.04, ROS for information transfer, the official Prophesee ROS driver and OpenEB 4.6.2 for event stream acquisition, and the NxSDK 1.0.0 library from Intel’s Neuromorphic Computing Research Team<ref:2610.21978#pg6>. Training is performed using PyTorch on an NVIDIA RTX 4090 GPU<ref:2610.21978#pg6>.

1. The integrated system can perform robust, low-power visual navigation by mimicking biological principles as described in Inspired by the event-driven, sparse, and asynchronous principles of biological visual systems. This allows for reliable perception during agile flight under severe resource constraints.

2. The system can achieve energy efficiency comparable to ANN methods while markedly reducing power consumption; specifically, the spiking perceptual module consumes only 19.57% of the power of its ANN counterpart and the spiking navigation module deployed on Intel Loihi consumes only 12.37% of that required by its ANN counterpart running on NVIDIA Orin NX.

3. The system can extract obstacle-relevant geometric features from the event stream, compensating for the lack of explicit spatial structure in raw events, which is achieved through a spiking depth estimation network followed by a spiking feature extraction network.

4. The navigation policy can learn efficiently using guidance learning and population coding, enabling it to perform action prediction based on encoded states, as detailed by the use of population coding for spike trains and the resulting action prediction via a trainable linear readout over the output firing rates.

5. The system can generate physically meaningful trajectories by optimizing a loss function that includes smoothness, safety, and goal-reaching terms: The trajectory loss Ltra consists of three terms: the smoothness loss Ls, the safety loss Lo, and the goal loss Lg.

6. The system can be trained end-to-end using a simulation-based training pipeline that utilizes losses like collision loss, smoothness loss, and goal-reaching loss, enabling joint optimization of all network modules.

7. The system can operate in real-world environments by utilizing a dual sensor setup for fine-tuning, where the depth supervision is achieved through the combination of the Prophesee EVK4 event camera and an Intel RealSense D435i to establish pixel-level correspondence between them.