Onboard Lightweight LLM Reasoning for Human-Aware Safe UAV Navigation

The gist The authors present a modular, fully onboard pipeline that closes the gap between conventional geometric perception and human-aware safety by turning visual observations into context-dependent safety constraints for real-time UAV navigation.

Perception

The perception system converts RGB-D frames into semantic and spatial primitives The shared BLIP encoder drives scene captioning and text-conditioned segmentation, which is fused with depth to localize obstacles in 3D. This process results in a list of (class, Pcenter, confidence) tuples at 10 Hz

Reasoning

The reasoning module is designed to infer obstacle-specific risk by using a lightweight LLM (Qwen-2.5-0.5B) stabilized with retrieval-augmented generation and grammar-constrained decoding. A key feature is the LLM's operation: it reasons once per object category rather than per frame, with its verdicts cached per class so previously seen categories are resolved without inference. For unknown classes, hybrid retrieval captures synonyms and anchors high-priority keywords in a local safety knowledge base. The output is constrained to a planner-compatible JSON schema using a GBNF grammar, which substantially reduces malformed generations.

Action

The action subsystem manages persistent obstacle identities and occlusion holding. Each detection is assigned a stable object id by greedy nearest-neighbor association within its class, and the planner map keeps the last-seen time and last known box for each id. An object whose age is below a hold time Th (3 s) stays in the map at its last position, and only afterwards are its voxels released. Unsafe regions in the planner are defined by an axis-aligned box Bi of side bbox size around Pcenter, and the unsafe set is B = S i Bi. The planner interface is intentionally minimal so that the back end can be swapped.

Feasibility and Performance

The system demonstrates onboard feasibility on a Jetson Orin NX, as the perception pipeline runs at 10 Hz while fallback LLM queries take approximately 0.8 s and remain off the control-critical path. The modular design reduces language-model input tokens by more than 19× relative to compared monolithic small VLMs. In urban navigation among pedestrians and vehicles, collision events per run drop from 1.35 to 0.45 (p=0.038) when using the full semantic pipeline with persistent ids and Th=3 s. Furthermore, in the semantics-aware landing case study, the full RAG-stabilized pipeline achieved a success rate of 96.0% among moving actors, compared to 34.0% for baseline geometric avoidance.

Limitations and Future Work

The research notes that the 224-pixel encoder input limits reliable small-object perception at range, particularly for pedestrians. The authors state that conservative pedestrian margins and short-term persistence mitigate this limitation during navigation. Future work will investigate higher-resolution perception within the same compute budget, improved perclass motion models for persistent objects, and cache-miss reasoning for novel categories. The evaluation also showed that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The paper concludes that a 0.5B-parameter LLM, grounded by retrieval, constrained to a machine-parseable schema, kept off the per-frame critical path by per-class verdict caching, and complemented by object persistence in the planner map. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that geometric avoidance alone reaches 34% among moving actors because it cannot anticipate that a pedestrian requires a larger, earlier safety margin than a static object. The unaugmented LLM is worse than no LLM at all (6–12%) because hallucinated or malformed verdicts destabilize the planner. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache. The full stack runs on a Jetson Orin NX and reduces collision events roughly threefold in urban navigation among pedestrians and vehicles—at the cost of longer paths and sharper maneuvers. The paper also shows that in the evaluated closed-loop scenarios, all observed categories were served from the warm verdict cache.

1. Semantic-to-Spatial Grounding Pipeline Implementation of 3D Safety Constraints: The system can convert RGB-D observations into obstacle-specific risk and 3D safety constraints. This allows for category-dependent clearance rather than treating all obstacles as generic occupied space, ensuring that a pedestrian receives different clearance than a parked car.

2. RAG-Stabilized, Grammar-Constrained Lightweight LLM Reasoning Module: This module can resolve unknown categories by querying a safety knowledge base via hybrid search and generating outputs restricted to the required schema, such as the verdict of whether an object is is dynamic or its bbox size. This is stabilized by ensuring per-class verdicts are cached so that the LLM reasons once per category rather than per frame.

3. Persistent Obstacle Identities with Occlusion-Aware Holding: The system can maintain a map where objects have stable IDs and their constraints are held through temporary occlusions, as it assigns each detection a stable object id by greedy nearest-neighbor association and keeps objects in the map for up to a hold time Th (3 s). This preserves category-specific safety semantics when planning around them.

4. Real-Time Onboard Feasibility via Modular Architecture: The system can operate on resource-constrained platforms like the Jetson Orin NX by decoupling a light vision-language encoder (VLE) from a text-only LLM and ground its output with depth. This allows the perception pipeline to run at 10 Hz while keeping the LLM queries off the control-critical path, ensuring real-time performance.

5. Safety Metric Improvement via Contextual Planning: The system can achieve significant safety gains, such as reducing collision events roughly threefold in urban navigation among pedestrians and vehicles, by injecting semantic constraints into a model-based planner that uses a heuristic augmented with a soft penalty h˜(p) = h0(p) + λ ϕ(p), leading to improved clearance.