Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mapping

arXiv:2605.17661 · cs.RO, cs.CV · Submitted 2026-05-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mapping".

Dev: The gist The system presents Mono-Hydra++, a real-time monocular RGB+IMU pipeline for indoor metricsemantic mapping and hierarchical 3D scene graph construction;

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at the paper "Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for three dee Indoor Mapping <ref:2605.17661#pg1,Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning>." This is about getting robots to not just see the shape of a room, but to actually understand what's in that room—objects, rooms, and their relationships.

Dev: It’s tackling the problem where standard metric maps are good for knowing where you are, but they don't tell you if there’s a chair or a table over there. This paper proposes a system that builds those scene graphs using only monocular RGB and IMU data, which is perfect for robots that can't carry big LiDAR sensors.

Taro: So the core idea here seems to be using multi-task learning to get both depth and semantics from just one camera, and then fusing those predictions in a way that keeps things stable over time.

Rosa: Exactly. The paper suggests they use a model called M2H-MX, which combines DINOv3 for vision with a Mamba decoder to create these dense depth and semantic outputs simultaneously. This allows them to share information across different tasks, which is key for this whole approach >

Dev: And they’re not just doing one thing per frame; they are integrating these predictions directly into the visual-inertial odometry pipeline, using predicted depth as sparse constraints for the VIO process >

Taro: That sounds like it addresses that issue where frame-wise predictions cause temporal flicker, because instead of just trusting what’s in each single picture, they are aligning those depth and semantic predictions using the VIO poses first >

Rosa: Right. And they do this by doing a "Pose-aware temporal fusion," which aligns recent depth and semantic predictions using the VIO poses before they actually fuse them into the final volumetric map >

Dev: That alignment step is important because it enforces consistency through explicit geometric warping instead of just updating weights online, which helps produce more stable object-level localization in that three dee scene graph > <ref:2605.17661#pg1>

Taro: So when you look at their improvements, they mention several specific things that make this pipeline better than previous work. One is using sparse predicted-depth factors for VIO improvement >

Rosa: That's a good starting point. They use "sparse predicted-depth factors" sampled from M2H-MX at tracked keypoints to give the VIO update metric anchors, which helps stabilize the robocentric VIO update as described in Equation (twenty-nine) > <ref:2605.17661#pg2>

Dev: And they also have semantic masking for dynamic scene robustness, which removes keypoints and sparse depth factors on classes likely to violate the static-scene assumption, like a person >

Taro: That’s interesting because it shows they aren't just relying on whatever the model guesses for every pixel; they are actively filtering out things that shouldn't be part of the map structure >

Rosa: And then they have this pose-aware temporal alignment for stable fusion, which we talked about. It uses VIO poses and intrinsics to align a short window of depth and semantic predictions to the current frame, which reduces temporal flicker >

Title and authors: Dev: That temporal warping process retains only the warped predictions that are depth-consistent with the current frame and not assigned to dynamic classes before fusing them across the window TK >

Taro: So when you think about what this means for a robot operating in a real room, it’s that you get a structured representation of the world—building, room, place, and object nodes >

Rosa: Exactly. This hierarchical three dee scene graph construction allows for higher-level reasoning beyond just raw geometry; it lets the robot understand spatial relations like "the chair is in the corner of this room" > <ref:2605.17661#pg1,hierarchical 3D scene graph construction>

Dev: The paper shows that they achieve one point six percent lower average trajectory error than the strongest RGB-D baseline on ScanNet evaluation, and in a real ITC building deployment using RealSense RGB+IMU, it improves average ATE by twenty-nine point eight percent over the strongest competing calibrated baseline >

Taro: So for someone only listening to this, what’s the real change here? It means monocular RGB plus IMU can support scene-graphlevel spatial understanding in a lightweight, deployable mapping pipeline for robotic platforms where RGB-D or LiDAR sensing is undesirable or unavailable >

Rosa: That's the big implication. It shows that you don't need that expensive sensor to get a rich semantic map; monocular RGB plus IMU can do it on resource-constrained platforms >

Dev: The performance metrics are quite strong on ScanNet, achieving forty-four point nine six global mIoU and thirty-three point eight one Box F1@zero point two five >

Taro: But the authors did flag a limitation, and that’s important to mention. They say the main remaining limitation is object-level semantic preservation, especially for lower-IoU object categories in classwise evaluation >

Rosa: That means while they get good at mapping rooms and large structures, identifying smaller or less obvious objects is where it struggles a bit >

Dev: Also, the temporal pose-warp fusion depends heavily on the quality of the VIO trajectory, and if pose estimates drift or dynamic objects occupy the scene for a long time, that evidence in the window can become outdated >

Taro: So future work they mentioned is about open-set and uncertainty-aware semantics to handle unseen categories, and also graph-level correction to recover missing object nodes or connect textual queries to objects >

Rosa: It's moving towards a system that not only maps the geometry but can reason about what those objects *are* in a much more robust, real-world way >

Dev: So there you have it. The Mono-Hydra++ paper shows how combining M2H-MX perception with a SuperPoint-assisted VIO front-end, coupled with these temporal alignment techniques, creates a unified pipeline for three dee scene graph construction using only monocular RGB and IMU input > <ref:2605.17661#pg1>

Taro: It’s a solid framework for getting metric semantic mapping going on platforms where the hardware budget is tight >

Rosa: We'll be ready to talk about the next paper when we come back, but for now, that’s what Mono-Hydra++ was all about.

The paper's summary: Rosa: So, essentially, this paper introduces Mono-Hydra++, which is this new pipeline for building three dee scene graphs just from a single camera feed and an IMU sensor, without needing that expensive RGB-D or LiDAR stuff you usually have to carry around.

Dev: Right. It’s about getting a structured representation of the indoor space—not just a point cloud, but nodes for buildings, rooms, and objects—using only what’s in the camera and the motion sensor.

Taro: The core mechanism is this multi-task model called M2H-MX that spits out both depth estimates and semantic labels simultaneously.

Rosa: Exactly. And they’re not just throwing those predictions into a map; they're coupling them directly into the visual-inertial odometry loop, using predicted depth as metric anchors to stabilize the robot’s movement estimation.

Dev: That coupling is key because it means when the robot moves, it uses those depth guesses to keep its pose estimates from getting completely lost, which is a big deal for real-time systems.

Taro: And they address a problem where frame-by-frame predictions can look jumpy or flickering; they use this pose-aware temporal alignment to smooth out that depth and semantic fusion across multiple frames using the IMU poses.

Rosa: So, what this means in practice is that you’re not just getting a messy set of coordinates; you’re building a consistent, metric-semantic world model on your robot while it moves through a house or an office.

Dev: The numbers show they achieve about one point six percent lower average trajectory error than the best RGB-D baselines on ScanNet, and in real testing with a RealSense camera, it’s about thirty percent better than the strongest existing calibrated system.

Taro: That performance is good for indoor mapping because it shows that this monocular approach can actually deliver spatial understanding that rivals systems using dedicated depth sensors.

Rosa: But they did flag something important: while they get great at mapping large structures and buildings, identifying smaller objects or specific categories within a class sometimes falls short.

Dev: That limitation is real; the object-level semantic preservation isn't perfect for every single tiny thing you want to track.

Taro: And then there’s the dependency on the VIO trajectory itself; if your motion estimate gets way off, that temporal alignment step can start reinforcing bad depth and semantic guesses over a longer window.

Rosa: So, this pipeline is lightweight enough to run fast—twenty-five frames per second on a Jetson Orin—but it still has those trade-offs between speed and perfect object classification.

Dev: What this means for the world is that we can deploy mapping and reasoning capabilities on much smaller, less expensive robots that don't need bulky LiDAR setups for basic indoor tasks.

Taro: It opens the door for more flexible, agile platforms because you can get a rich spatial understanding without tying yourself to a specific sensor type.

Rosa: Next time we talk, we’ll look at how they tackle those uncertainties in their loss weighting and see how that affects real-world reliability.

The paper's improvements: Taro: So we're looking at what they actually improved in their pipeline—they’ve introduced several specific technical fixes to make it more robust in the real world than just the initial design.

Rosa: Right, so beyond just having a depth and semantic head, they added things like using sparse predicted-depth factors for the VIO front-end.

Dev: That’s a big deal because those sparse depth factors act as metric anchors to stabilize that robocentric VIO update, which is crucial when you’re running tight real-time loops.

Taro: And they also put in semantic masking, which is really smart for robustness; it tells the system to ignore keypoints or depth factors on classes likely to be dynamic, like a person walking by.

Rosa: That means if the world misbehaves and something moves quickly, the mapping stays more stable because it’s filtering out those noisy inputs before they mess up the trajectory.

Dev: Then there's this pose-aware temporal alignment we talked about earlier; they use the VIO poses to align a short window of depth and semantics before fusing them, which really cuts down on that temporal flicker you get when things jump around frame to frame.

Taro: That warping process is clever because it only keeps the predictions that are geometrically consistent with where the robot actually moved, not just whatever the model guessed randomly.

Rosa: It leads directly to this hierarchical three dee scene graph construction they mention; they’re building nodes for buildings, rooms, and specific objects instead of just a flat map.

Dev: That structure is what makes it useful for higher-level reasoning; you can ask questions about the room layout or object relationships rather than just seeing raw geometry.

Taro: They also refined their perception using a Cross-Task Mixer module, which selectively injects complementary cues from related tasks to help the depth and semantic predictions become more accurate overall.

Rosa: That means they’re not relying on one single model output; they’re letting different parts of the system help each other improve the final result.

Dev: And before that, they tweaked their loss function to use learned uncertainty weighting, using a formula involving sigma squared to balance how much weight we give to the depth and semantic losses.

Taro: That is designed specifically to improve prediction reliability before they even get to the geometric alignment stage, making sure we trust the inputs more.

Rosa: So what this changes for us is that while it’s still a monocular system, you’re getting much higher quality trajectory accuracy and a more reliable semantic map for navigating complex indoor environments.

Dev: The caveat they raise again is that the temporal pose-warp fusion still needs a good VIO trajectory to work well, so if the motion estimation drifts too much, the fusion can start introducing errors.

Taro: That’s true; longer windows of data are more susceptible to outdated information if the robot gets lost in a loop or encounters dynamic changes.

Rosa: So, while it’s a solid pipeline for deployment on resource-constrained platforms, you still need a high-quality motion estimate to keep that semantic structure consistent over time.

Dev: Next up, we’re going to look at the evaluation results showing exactly how much better this is compared to the systems they tested against on ScanNet and in real building deployments.

Conclusion: Rosa: So to wrap up, Mono-Hydra++ is this unified pipeline that uses M2H-MX perception coupled with a SuperPoint-assisted VIO front end to build a metric semantic scene graph from just monocular RGB and IMU data.

Dev: It means you get real-time three dee mapping and scene understanding without needing that expensive RGB-D or LiDAR hardware, which is a huge practical win for robotics.

Taro: The real strength is the pose-aware temporal alignment; it makes the depth and semantic fusion much more stable over time by aligning predictions with the actual robot motion.

Rosa: And they showed solid numbers, getting about one point six percent lower trajectory error than strong RGB-D baselines on ScanNet, which is a good sign for real world testing too.

Dev: The speed is also there; they got it running at twenty-five frames per second on a Jetson Orin NX, which shows this thing is feasible for onboard spatial understanding.

Taro: But we still have those caveats—the object-level semantic preservation isn't perfect for smaller categories, and the temporal alignment relies heavily on a good VIO trajectory to work well.

Rosa: Exactly. The authors admit that while it maps large structures reliably, identifying tiny objects is where they still struggle a bit in the classwise tests.

Dev: And if the motion estimate drifts significantly, that temporal fusion can start reinforcing inaccurate depth and semantic guesses over longer time windows.

Taro: Looking ahead, they mentioned future work focusing on open-set semantics to handle unseen categories and graph-level correction to recover missing nodes in the scene graph.

Rosa: That points toward a system that can reason about the world in a much more open way than just the fixed classes they trained on.

Dev: It’s moving toward something that connects those three dee structures with language, which is where we see some of the most interesting new directions in autonomy.

Taro: So, for people who only listen to this show, think of Mono-Hydra++ as a lightweight way to get a structured spatial understanding on a robot using just a camera and motion sensor.

Rosa: It’s definitely a solid framework for getting metric semantic mapping going on platforms where the hardware budget is tight.

Dev: We’ll keep an eye on how they address those uncertainty issues in the next steps of their research.

Department of Earth Observation Science, University of Twente

cs.RO, cs.CV

Submitted: 2026-05-17

Updated: 2026-10-19

Comments: Submitted to ISPRS Journal of Photogrammetry and Remote Sensing. 50 pages, figures and tables included. Code: https://github.com/BavanthaU/mono-hydra-pp.git

Code: https://github.com/BavanthaU/mono-hydra-pp

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 79/100

The gist: The gist The system presents Mono-Hydra++, a real-time monocular RGB+IMU pipeline for indoor metricsemantic mapping and hierarchical 3D scene graph construction; this makes it suitable for agile

Key concepts

M2H-MX: This is the core perception model that uses a DINOv3 backbone combined with a Mamba decoder. It is trained to simultaneously predict both the 3D depth of the scene and its semantic labels (what objects are present). This dual prediction makes it highly effective for understanding both geometry and object identity in images.
Geometry-aware perception-to-VIO coupling: Instead of just using depth predictions as raw data, this method integrates them directly into the visual-inertial odometry (VIO) process. The predicted depth provides crucial metric constraints for the motion estimation, while semantic masks help filter out unreliable areas during movement tracking.
Pose-aware temporal alignment: This technique aligns recent depth and semantic predictions using the calculated VIO poses before combining them into a single map. By warping predictions based on accurate pose information, it prevents visual 'flicker' and ensures that the final metric-semantic scene graph is temporally consistent across different frames.

Terminology

Graph Construction — 3× in this paper · explained in 3 episodes

Summary

The gist The system presents Mono-Hydra++, a real-time monocular RGB+IMU pipeline for indoor metricsemantic mapping and hierarchical 3D scene graph construction; this makes it suitable for agile robotic platforms where RGB-D or LiDAR sensors are impractical

How it works

The system combines M2H-MX, a DINOv3-based multi-task model for depth and semantics, with a deep-feature-based visual-inertial odometry (VIO) front-end to achieve real-time monocular RGB+IMU pipeline for indoor metricsemantic mapping and hierarchical 3D scene graph construction. This involves several key components that address the challenges of monocular scene graph construction.

The key advances introduced by Mono-Hydra++ are:

(Lightweight monocular metric-semantic mapping)

Mono-Hydra++ reconstructs metrically scaled semantic meshes and hierarchical 3D scene graphs from monocular RGB plus IMU input, avoiding the RGBD or LiDAR dependency of existing real-time scene graph pipelines while retaining structured spatial representations suitable for indoor mapping and robotic reasoning.

(Geometry-aware perception-to-VIO coupling)

Mono-Hydra++ integrates M2H-MX predictions into the visual-inertial mapping pipeline rather than using them only as frame-wise dense outputs, where predicted depth provides sparse metric constraints for VIO, while semantic masks reduce the influence of dynamic or unreliable regions during motion estimation.

(Pose-aware temporal fusion for stable scene graphs)

MonoHydra++ aligns recent depth and semantic predictions using VIO poses before volumetric fusion, which reduces temporal flicker, improves metricsemantic consistency, and produces more stable object-level localization in the resulting 3D scene graph.

M2H-MX Perception Model

The M2H-MX [14] architecture couples a frozen DINOv3 [48] transformer backbone with a Mamba-based decoder built from Mamba blocks [49] together with efficient convolutional and linear layers. This model constructs shared multi-scale features, refines them through sceneconditioned decoding and controlled cross-task interaction, and produces the dense outputs used later for temporal stabilization and mapping. The HFA adapter converts DINOv3 token outputs into multi-scale feature maps for dense decoding. The Register-Gated Mamba (RGM) block updates each pyramid scale by injecting global context through the register vector r and modeling spatial dependencies with Mamba blocks [49].

Task Heads and Losses

The final task heads convert the refined decoder features into dense outputs for depth and semantics. The Depth Head uses an adaptive binning principle introduced in AdaBins [50] to predict per-pixel depth relative to an image-dependent partition. Semantic logits are produced by a lightweight convolutional predictor. The supervision stage uses task-specific losses, such as cross-entropy (CE) for categorical semantics and the scale-invariant logarithmic loss (SILog) for scaleambiguous monocular depth. Uncertainty weighting is applied to balance these tasks using learned uncertainty weighting [51].

SuperPoint-Assisted VIO Front-End

The system uses a classical robocentric VIO module following an RVIO2-style square-root formulation with QR-based updates [30]. SuperPoint replaces the hand-crafted detector and descriptor, and M2H-MX depth provides sparse metric anchors. Sparse depth factors are sampled from M2H-MX at tracked keypoints to form sparse depth factors, which are then gated by angular-velocity, reprojection, and semantic checks. Semantic masking removes keypoints and sparse depth factors on classes that are likely to violate the static-scene assumption.

Temporal Alignment and Fusion

Predicted depth Dˆt is refined through consistency with VIO pose updates, where its 3D position is predicted as Pt(x) = Π−1(x, Dˆt(x)) and propagated to the next frame via Pt→t+1 = Tt→t+1 Pt. Temporal Pose-Warp Alignment aligns a short temporal window of depth and semantic predictions to the current frame using VIO poses and intrinsics, which enforces consistency through explicit geometric warping rather than online weight updates. This process retains only warped predictions that are depthconsistent with the current frame and not assigned to dynamic classes before fusing them across the window TK.

Evaluation Results

Mono-Hydra++ achieves 1.6% lower average trajectory error (ATE) than the strongest RGB-D baseline in our comparison on the Go-SLAM ScanNet evaluation subset. In a real ITC building deployment using RealSense RGB+IMU, it improves average ATE by 29.8% over the strongest competing calibrated baseline. The system shows strong performance on ScanNet, achieving 44.96 global mIoU, 42.59 Radius F1@0.5m, and 33.81 Box F1@0.25. In the ITC real-world test setting, the ONNX deployment reaches 25.53 FPS end-to-end on a Jetson Orin NX 16GB. The results indicate that monocular RGB+IMU can support scene-graphlevel spatial understanding in a lightweight, deployable mapping pipeline for robotic platforms where RGB-D or LiDAR sensing is undesirable or unavailable.

Discussion and Limitations

The main remaining limitation is object-level semantic preservation, as large structural classes and large objects are recovered more reliably than the lower-IoU object categories in the classwise evaluation. Temporal pose-warp fusion depends on the quality of the VIO trajectory, and longer windows can introduce outdated semantic evidence when pose estimates drift or dynamic objects occupy the scene.

Conclusion

Mono-Hydra++ presents a unified monocular RGB+IMU pipeline for real-time 3D scene graph construction. The system uses M2H-MX as its dense depth and semantic perception module and couples it to a SuperPointassisted, RVIO2-style robocentric VIO front-end. Sparse predicted-depth factors and semantic-aware filtering improve the VIO update, while the resulting odometry and keyframe pose graph provide the messages required by the Mono-Hydra/Hydra backend for metric-semantic mesh construction, loop-closure proposal handling, backend graph optimization, and hierarchical 3D scene-graph generation. Pose-aware temporal alignment further stabilizes depth and semantic fusion across frames. Across NYUDv2 and Cityscapes, M2H-MX improves multi-task depth and semantic prediction, and within Mono-Hydra++ these gains translate into stronger trajectory accuracy, competitive 3D reconstruction quality, and improved semantic mesh and object-level scene graph quality on ScanNet and 7-Scenes using monocular RGB+IMU input. Taken together, the results show that real-time monocular RGB+IMU can support scene-graphlevel spatial understanding in a lightweight, deployable mapping pipeline for robotic platforms where RGB-D or LiDAR sensing is undesirable or unavailable.

Future Work

Future work will extend the framework in three directions: open-set and uncertainty-aware semantics to handle previously unseen categories while preserving small-object evidence under noisy dense predictions, graph-level correction and language grounding to recover missing object nodes and connect textual queries to objects, rooms, and relations in the scene graph, and downstream navigation on resource-constrained platforms where the scene graph supports planning and decision making under limited onboard compute.

Improvements for AI systems

Bold header: Sparse predicted-depth factors for VIO improvement

The system can now improve pose estimation by using sparse predicted-depth factors sampled from M2H-MX at tracked keypoints, which provides metric anchors to stabilize the robocentric VIO update, as described in Equation (29).

Bold header: Semantic masking for dynamic scene robustness

The improved system can enhance motion estimation consistency by employing semantic masking, which removes keypoints and sparse depth factors on classes likely to violate the static-scene assumption, such as person in the reported experiments.

Bold header: Pose-aware temporal alignment for stable fusion

The system will achieve more stable metric-semantic fusion across time by using pose-aware temporal alignment, which aligns recent depth and semantic predictions using VIO poses before volumetric fusion, thereby reducing temporal flicker.

Bold header: Hierarchical 3D scene graph construction

The pipeline can now generate a structured representation by building a hierarchical 3D scene graph with nodes for building, room, place, and object, enabling higher-level reasoning beyond raw geometry.

Bold header: Task-specific feature refinement via Cross-Task Mixer (CTM)

The system can produce more accurate depth and semantic features by using the Cross-Task Mixer (CTM) module to selectively inject complementary cues from related tasks while suppressing harmful interference, as detailed in Equation (12).

Bold header: Adaptive binning for metric depth estimation

The depth head can now estimate per-pixel depth with adaptive resolution by utilizing the adaptive binning principle introduced in AdaBins [50], allowing the model to allocate depth resolution according to the scene content.

Bold header: Uncertainty-aware loss weighting

The system can improve prediction reliability before geometric alignment by using learned uncertainty weighting where losses are weighted as L = X a / (2σ 2 a) La + log σa, which is designed to improve prediction reliability.

Bold header: Real-time deployment feasibility

The perception component can be deployed on resource-constrained platforms by running the M2H-MX model at 25.53 FPS on a Jetson Orin NX 16GB, demonstrating that the proposed monocular perception component is suitable for onboard spatial understanding.

Abstract

Autonomous agile robots need more than metric geometry: they must understand objects, rooms, places, and spatial relations for search, inspection, exploration, and human-robot interaction. Conventional metric maps support localization and collision avoidance, but do not provide this semantic and relational structure. 3D scene graphs address this gap by connecting geometry with object-level and room-level understanding. Building such representations on agile platforms remains difficult because aerial and lightweight robots operate under strict payload, power, and compute limits, making RGB-D cameras and LiDAR sensors impractical for many onboard settings. We present Mono-Hydra++, a real-time monocular RGB plus IMU pipeline for indoor metric-semantic mapping and hierarchical 3D scene graph construction. The system combines M2H-MX, a DINOv3-based multi-task model for depth and semantics, with a deep-feature visual-inertial odometry front end, sparse predicted depth constraints in the VIO-derived pose graph, semantic masking for dynamic regions, and pose-aware temporal alignment before volumetric fusion in the Mono-Hydra backend. On the Go-SLAM ScanNet evaluation subset, Mono-Hydra++ achieves 1.6% lower average trajectory error than the strongest RGB-D baseline in our comparison, while using only monocular RGB plus IMU input. On calibrated 7-Scenes, it improves average ATE by 29.8% over the strongest competing calibrated baseline. We further validate Mono-Hydra++ in a real ITC building deployment using RealSense RGB plus IMU and demonstrate embedded feasibility by deploying the ONNX/TensorRT FP16 M2H-MX-L perception model at 25.53 FPS on a Jetson Orin NX 16GB. These results show that Mono-Hydra++ can provide real-time metric-semantic mapping and scene graph construction for resource-constrained robotic platforms without relying on active depth sensors.