LifelongCrossNav: Persistent 3D Semantic Memory for Cross-Floor Multi-Object Navigation

Summary of LifelongCrossNav: Persistent 3D Semantic Memory for Cross-Floor Multi-Object Navigation

This paper addresses the combined challenges of sequential multi-object Object-Goal Navigation (ObjectNav) and cross-floor navigation in unknown indoor environments. The authors note that existing research treats these problems separately: Cross-floor methods model stairs, floor transitions, or floor-level reasoning, but generally consider only one object goal, while Multi-object navigation instead retains environmental information across a sequence of object goals but primarily rely on planar representations or avoid goal sequences that require stair traversal. They observe that Planar maps may collapse vertically overlapping spaces, while single-object cross-floor methods need not preserve queryable semantic observations for future object goals, concluding that A unified solution must therefore support both vertical traversability and persistent semantic memory.

Proposed Framework: LifelongCrossNav

To fill this gap, the authors present LifelongCrossNav, a framework for sequential multi-object ObjectNav in unknown multi-floor indoor environments. Within each episode, the agent receives an ordered sequence of K object goals, where the agent receives only the current object goal gk; the next object goal gk+1 is revealed and activated only after the current object-goal subtask has been successfully completed. The framework maintains a persistent sparse 3D voxel memory that incrementally accumulates geometric structure, traversability states, and vision-language features, allowing subsequent object-goal queries to retrieve previously acquired scene information without rebuilding the map.

The method consists of several key components:

1. Support-Aware 3D Voxel Mapping: The environment is represented using a sparse 3D voxel map that combines support-aware geometry with goal-independent vision-language features. This representation preserves height and distinguishes vertically overlapping rooms, corridors, landings, and staircases. The map abstracts observed space into functional voxel types including Occupied, Traversable, Stair, and Unsupported. Free space with reliable support is classified as Traversable, whereas free space without observed support is classified as Unsupported and provides geometric evidence for potential downward transitions. Stair voxels are confirmed jointly by semantic and geometric observations, using SegFormer-B2 to extract pixel-level stair masks that are verified through local height variation, step trends, spatial continuity, and support relationships.

2. Persistent 3D Semantic Memory: The system stores goal-independent vision-language features in the same sparse 3D coordinate system. Using SED-based dense vision-language encoding, each RGB observation produces a spatial feature map aligned with the CLIP text-embedding space. Pixel features are lifted to the 3D surface voxels and fused across multiple views using a cumulative weighted average. When an object goal is activated, its text embedding is compared with the stored surface-voxel features using cosine similarity, generating a query-conditioned 3D similarity field without rebuilding the geometric or semantic map. High-response historical regions are clustered and retained as History POIs.

3. Unified Navigation Policy: The policy coordinates same-floor frontier exploration, live and historical point-of-interest (POI) retrieval, stair navigation, and target-object search. It selects among Basic Frontiers, Stair Frontiers, History POIs, and Live POIs, supporting three navigation modes: Basic Explore, Stair Explore, and POI Navigation. The hierarchical switching prioritizes Reliable current target evidence and reachable POIs... over unexplored geometric candidates. Different modes use a common 3D A* planner with different admissible voxel types and graph connectivity. The final approach requires verification through detection confidence, mask quality, visibility, viewing direction, and 3D distance, where Reaching either a History POI or a Live POI does not by itself complete the subtask.

New Benchmark: HM3D-MFMON

The authors introduce HM3D-MFMON, a benchmark for sequential Multi-Floor Multi-Object Navigation built on HM3D scenes. It contains 927 three-goal episodes from 36 multi-floor HM3D scenes, including 288 Cross-Floor-Required episodes — episodes that cannot be completed without at least one floor transition. The benchmark covers six target object categories: chair, bed, toilet, plant, sofa, and tv monitor.

Post-hoc Evaluation Protocol

A key contribution is the post-hoc stage-wise shortest-path protocol. The authors explain that The start of a later subtask depends on the target object instance selected for the previous object goal and on the position from which that subtask was completed, and a globally optimized route uses future object goals that have not yet been issued, violating the causal information constraint. Instead, for each stage, the reference distance uses the actual agent position when the k-th object goal is activated and computes the minimum geodesic distance to valid navigable positions of the current target category. They report Conditional SR and Conditional SPL metrics that evaluate success and efficiency only among episodes that reach stage k, which is particularly useful for examining whether later object goals are reached more efficiently as geometric and semantic memory accumulates.

Experimental Results

• Multi-object navigation: LifelongCrossNav achieves sequence-level SR/SPL/PR/PPL of 29.13/9.64/48.40/16.70 on all 927 episodes, substantially outperforming the OneMap baseline's 16.83/6.95/33.69/14.03. On the 288 Cross-Floor-Required episodes, OneMap cannot complete any full Cross-Floor-Required sequence (SR = 0.00), while LifelongCrossNav achieves SR = 7.99, demonstrating that persistent planar memory alone is insufficient when task completion requires explicit stair traversal and cross-floor connectivity.

• Ablation results: A variant w/o H-POI that disables History POI generation achieves slightly higher SR (29.77) but lower SPL (8.46) and PPL (14.91). The authors attribute the SR difference to a limited number of outcome reversals after History POI retrieval changes the selected target-object instance or approach viewpoint, with additional misdetections concentrated primarily on bed, where YOLOv7 occasionally confuses beds with visually similar sofas. They conclude the consistent gains in SPL, PPL, and later-stage Conditional SPL... provide clearer evidence of improved path efficiency, while also highlighting target verification as the main remaining source of error.

• Single-object navigation: As an auxiliary evaluation on the standard HM3D ObjectNav validation split, LifelongCrossNav achieves competitive SR of 59.7 and SPL of 28.0, retaining competitive success without online LLM reasoning.

Conclusion

The authors summarize: "LifelongCrossNav combines support-aware 3D voxel mapping, persistent vision-language semantic memory across object goals, and a unified policy for planar exploration and stair navigation, enabling the agent to jointly exploit vertical structure, cross-floor connectivity, and historical semantic observations. Experimental results show consistent improvement over the planar baseline, and Ablation results show that History POIs reduce repeated exploration and improve later-goal efficiency. Future work will study real-world deployment and more robust cross-floor navigation."

• The AI navigation agent can maintain a persistent sparse 3D voxel memory across a sequence of object goals, accumulating geometry, traversability states, and vision-language features without rebuilding the map when a new goal is issued.

• It can explicitly represent vertical structure: marking voxels as Occupied, Traversable, Stair, or Unsupported, allowing it to distinguish overlapping rooms, corridors, landings, and staircases that planar maps collapse.

• It can detect staircases by combining semantic segmentation with geometric verification (height variation, step trends, spatial continuity, support relationships), enabling reliable cross-floor transitions in unknown multi-floor buildings.

• It can store SED-based vision-language features in 3D surface voxels, aligned with CLIP text embeddings, so when a new object goal is activated it can compare the text embedding to all stored voxel features via cosine similarity and generate a query-conditioned 3D similarity field.

• It can cluster high-response historical regions into History POIs, allowing the agent to revisit previously observed candidate locations for the current object goal without re-exploring, reducing repeated traversal and improving later-goal efficiency.

• It can unify navigation modes — basic frontier exploration, stair frontier exploration, live POI navigation, and history POI navigation — under a single hierarchical policy that prefers reliable current-goal evidence and reachable POIs over unexplored geometric candidates.

• It can plan paths across floors using a common 3D A* planner with support-aware voxel types and appropriate graph connectivity, allowing stair traversal and cross-floor connectivity to be integrated into the same planning framework.

• It can verify whether a target is actually present before completing a subtask, using detection confidence, mask quality, visibility, viewing direction, and 3D distance, rather than treating arrival at a POI as success — reducing false positives and improving sequence completion.

• It can evaluate navigation progress stage-by-stage with a causally valid post-hoc shortest-path protocol: each stage’s reference distance is computed from the agent’s actual position when that goal is activated, using only currently available information, producing Conditional SR and Conditional SPL metrics that isolate whether later goals become easier as memory accumulates.

• The resulting system can complete sequential multi-object object-goal navigation in unknown multi-floor environments, including episodes that require at least one stair traversal, while retaining competitive performance on single-object ObjectNav without online LLM reasoning.

Object-goal navigation has made substantial progress in semantic perception and exploration, yet persistent memory for multi-object navigation and cross-floor navigation are still commonly addressed separately. We present LifelongCrossNav, a framework for sequential multi-object ObjectNav in unknown multi-floor indoor environments. Within each episode, the agent receives an ordered sequence of object-goal queries while continuously maintaining a shared sparse 3D semantic voxel memory. This memory incrementally accumulates geometric structure, traversability states, and vision-language features, allowing subsequent object-goal queries to retrieve previously acquired scene information without rebuilding the map. To support persistent search across floors, LifelongCrossNav combines support-aware 3D traversability mapping, stair-specific perception, and direction-aware stair traversal. A unified navigation policy coordinates same-floor frontier exploration, live and historical point-of-interest retrieval, stair navigation, and target-object search and approach. We further introduce HM3D-MFMON, a benchmark for sequential Multi-Floor Multi-Object Navigation built on HM3D scenes, including a dedicated subset in which completing the full sequence of object-goal subtasks requires at least one floor transition. Experimental results show that LifelongCrossNav consistently outperforms a representative planar persistent semantic-map baseline on HM3D-MFMON, demonstrating that persistent 3D semantic memory and cross-floor traversability modeling effectively support sequential multi-object navigation in multi-floor environments. Project page: https://flageval-baai.github.io/LifelongCrossNavPage.