ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy

arXiv:2511.22250 · eess.IV, cs.CV · Submitted 2025-11-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy".

Jane: Here's the gist of the paper. The gist: ColonAdapter presents a self-supervised fine-tuning framework that adapts 3D geometric foundation models for colonoscopy geometry estimation by leveraging pretrained geometric priors while tailoring them to clinical data.

Tom: Problem and Motivation Estimating 3D geometry from monocular colonoscopy images is challenging due to "non-Lambertian surfaces, moving light sources, and large textureless regions" (Page 1).

Jane: While recent 3D geometric foundation models eliminate the need for multi-stage pipelines, their performance "deteriorates in clinical scenes" because they are primarily trained on natural scene datasets and struggle with "specularity and homogeneous textures typical in colonoscopy, leading to inaccurate geometry estimation" (Page 1).

Tom: Traditional feature-based reconstruction methods struggle because of "sparse and unevenly distributed key points in endoscopic images," resulting in poor reconstructions (Page 1).

Title and authors: Tom: So, we're looking at the paper titled "ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy." This work is about taking these big three dee geometric foundation models and making them actually work for colonoscopy images <ref:2511.22250#pg1>. It’s essentially a self-supervised fine-tuning framework that uses pre-trained geometric knowledge but customizes it for what happens inside a colonoscope.

Jane: That makes sense, Tom. So, the core idea is adapting these general models so they don't fail when they see things like those challenging surfaces and textureless spots we see in clinical colonoscopy scenes. It’s about leveraging those geometric priors while making sure the model understands the specific details of a procedure.

Lu: What's interesting here is that they are tackling the problem where models trained on natural scenes struggle with specularity and homogeneous textures, which are very common in these images, leading to inaccurate geometry estimation >

Meng: But how do they actually fix that without having perfect ground truth depth data available for every single scan? That's the practical hurdle I worry about when deploying anything like this.

Lalam: The paper proposes using only monocular videos and photometric loss supervision, which means they rely on the geometric priors of the foundation model to guide them rather than pixel-wise three dee point map losses > <ref:2511.22250#pg1>

Tom: Exactly. They don't need camera information or ground truth depth during training. But they introduce a Detail Restoration Module, or DRM, to help with fine details and a geometry consistency loss to keep the predicted geometry coherent across frames >

Jane: The DRM sounds like the key for handling those low-texture regions that we talked about earlier. It fuses fine details from an auxiliary convolutional encoder into the foundation model, which should help recover more accurate shapes >

Lu: And they also added a confidence map into the photometric loss for training stability, which means they are focusing on reliable areas during training, and they introduced a geometry consistency loss to address those scale inconsistencies >

Meng: From an engineering standpoint, that sounds like it adds complexity to the pipeline. How much overhead does this fine-tuning add to inference time when we actually need to use the model in a clinic?

Tom: Well, the ablation study showed that a lightweight sequential fusion strategy for the feature fusion in their DRM worked best, and removing those confidence-weighted photometric losses actually made performance degrade, even below their baseline >

Jane: So it seems they've tuned it down to be as efficient as possible while keeping the core improvements in place. This suggests a method that’s both robust and practical for clinical use >

Lu: And when you look at the results, they show strong performance across three different tasks: camera pose estimation, monocular depth estimation, and dense point map estimation on datasets like SimColthree dee and CSD >

Title and authors: Meng: That's good to hear about the numbers. What about real-world data? How does it hold up when we move from synthetic or simulated data to actual patient scans?

Tom: They did evaluate it on EndoMapper, a real-world dataset, and they showed improvements over other methods like DUSt3R, especially in those challenging conditions with specularity and textureless regions >

Jane: It sounds like they are not just tinkering with the model; they are specifically targeting the clinical problems that make existing models fall apart when they see real colonoscopy footage >

Lu: The point map estimation accuracy on SimColthree dee was one point seven four two, and their RMSE log was zero point one one one, compared to DUSt3R's forty-one point five three five accuracy and RMSE log of zero point four eight six >

Meng: Those are big differences in metrics there. It shows a significant improvement in how accurately they can reconstruct the physical shape of the colonoscopy image without needing those difficult ground truth intrinsic parameters >

Tom: And they even managed to estimate camera poses with an Absolute Translation Error of zero point zero zero six two on the SimColthree dee dataset, which was better than AF-SfMLearner's error of zero point zero zero six seven >

Jane: So to summarize, this paper, "ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopy," introduces a self-supervised fine-tuning framework that adapts foundation models for colonoscopy geometry estimation by leveraging pretrained geometric priors while tailoring them to clinical data >

Lu: It's about using photometric loss supervision on monocular videos, and then adding those key components: the Detail Restoration Module and the geometry consistency loss, alongside the confidence-weighted photometric loss >

Meng: And they’re showing that these specific additions are what really matter for robustness when dealing with clinical images rather than just using a raw foundation model >

Tom: So, to wrap up on this paper, ColonAdapter shows strong performance in pose estimation, depth prediction, and point map reconstruction across synthetic and real datasets >

Jane: It's an interesting approach because it tackles the scene challenges head-on by tailoring the foundation model to clinical data using these specific modules >

Lu: The implication is that we can take massive three dee geometric models and make them specialized tools for medical imaging tasks, provided we give them the right supervision during fine-tuning > <ref:2511.22250#pg1>

Meng: I'm just thinking about deployment. If this adaptation process is manageable, it opens up a path for putting these geometric models into actual clinical diagnostic workflows >

Tom: That’s the goal. The future work they mentioned involves integrating this adapted geometric foundation model into a SLAM framework for efficient reconstruction >

Jane: So we'll keep an eye on how that integration plays out as they move toward more complete reconstruction systems >

The paper's summary: Tom: So, we're looking at this summary of ColonAdapter which basically says they took these massive three dee geometric foundation models and tweaked them specifically for colonoscopy geometry estimation >

Jane: Right, so it’s not just using a general model; it’s self-tuning that model using only video data without needing any ground truth depth information during the training phase >

Tom: Exactly. They are leveraging those huge geometric priors already baked into the foundation model but they are tailoring them to handle things like those messy, textureless areas and specularity you see in real endoscopic images >

Lu: The core mechanism there is this self-supervised fine-tuning strategy driven by photometric constraints, which means they use how the pixels change between frames as their main supervisor >

Meng: That sounds clever because it bypasses needing perfect, expensive ground truth depth maps for every single scan during the adaptation process >

Tom: It does. And they introduce a Detail Restoration Module or DRM to specifically help recover those fine details that those general models usually miss when they look at complex anatomy >

Jane: So the DRM is essentially an extra layer of convolutional encoder fused into the foundation model just to boost reconstruction accuracy for high-frequency details >

Lu: And then they add this geometry consistency loss which uses terms like Tflow and Tpose to make sure that the predicted poses and point maps actually line up coherently across different video frames >

Tom: That’s crucial because without that coherence, you just get a bunch of weird, drifting geometry predictions when you try to reconstruct a three dee shape >

Jane: And they also use this confidence-weighted photometric loss which helps the training stay stable by focusing on the regions where the model is actually making sense >

Meng: I see how that would help with deployment because it makes sure we aren't training on bad data that just throws off our results when we try to use it in a real clinical setting >

Lu: The results showed strong performance across three major three dee tasks: camera pose estimation, monocular depth estimation, and dense point map estimation on synthetic and real datasets >

Tom: Look at the numbers for the point map accuracy on SimColthree dee, they got a one point seven four two accuracy and an RMSE log of zero point one one one >

Jane: That’s a significant jump compared to DUSt3R which had an accuracy of forty-one point five three five and an RMSE log of zero point four eight six >

Meng: The fact that it beats DUSt3R on SimColthree dee, especially in those tricky conditions like specularity, is what makes me pay attention >

Tom: They also managed a very tight Absolute Translation Error of zero point zero zero six two on the camera pose estimation for SimColthree dee which is better than the AF-SfMLearner error of zero point zero zero six seven >

Lu: And the ablation studies confirmed that adding those components—the feature fusion and those specific loss terms—actually improved performance, with removing them making things worse >

Jane: So, what this really means for us is we can take these huge three dee geometric models and specifically adapt them to handle the unique visual challenges of colonoscopy images without needing massive amounts of labeled depth data >

Tom: It shows that tailoring the foundation model with these specific modules and losses is a very effective way to make it useful in clinical imaging applications >

Lu: The authors are looking ahead by planning to integrate this adapted geometric foundation model into a SLAM framework for more efficient reconstruction later on >

The paper's improvements: Tom: So, to summarize these improvements, they're not just tweaking the existing model; they're introducing several new supervisory mechanisms to make this foundation model robust for clinical work >

Jane: Right, so it’s about adding specific ways to train it up—like using that confidence map in the loss and building in those geometric consistency checks >

Tom: They are basically saying that just relying on the raw geometric priors isn't enough for clinical scenes, so you gotta supervise them with photometric constraints and a geometry consistency loss >

Lu: The key idea is that this fine-tuning adapts the model to handle those messy things like non-Lambertian surfaces by making sure the predictions are geometrically sound across time frames >

Meng: That’s smart because if the geometry drifts frame to frame, any reconstruction you do from a video sequence is going to be garbage >

Tom: And they also added that Detail Restoration Module or DRM which fuses fine details from an auxiliary convolutional encoder into the foundation model >

Jane: So this DRM helps it recover those really small textures and fine anatomical features that are usually lost when the model tries to generalize across a whole scene >

Lu: That fusion strategy is what allows for enhanced multi-level detail extraction, which is what you need when you're trying to get accurate point maps in a complex region >

Tom: And they didn't just stop there; they showed that removing those specific loss terms—the confidence-weighted photometric loss and the geometry consistency loss—actually made the performance degrade, even below their baseline >

Jane: That tells you these additions aren't just extra complexity; they are actually necessary parts of the mechanism for it to work well >

Meng: From an engineering standpoint, that means we can be confident that if we implement this structure, the resulting model will perform better than a simpler adaptation approach >

Tom: And they also showed that a lightweight sequential fusion strategy for the feature fusion in their DRM was actually the best performer compared to more complex attention-based modules >

Lu: So it’s a balanced approach where you get that fine detail recovery without adding too much computational overhead during inference >

Jane: It seems like they've really nailed how to bridge the gap between a massive, general foundation model and a very specific, high-detail clinical application >

Tom: And when you look at the results again, the performance across camera pose estimation and monocular depth estimation was quite solid on those synthetic datasets >

Jane: So what this changes for someone listening is that we can expect these adapted models to be much more reliable than they would be if we just took a general model and threw it at a colonoscopy image without any of this specific fine-tuning >

Conclusion: Tom: So we’re wrapping up on ColonAdapter, which is this framework for adapting three dee geometric foundation models to colonoscopy geometry estimation >

Jane: It’s really about showing that you can take these massive general models and fine-tune them using only video data to get much better results for specific clinical tasks >

Tom: They proved that by supervising the photometric loss and adding those specific modules, like the DRM, you can handle those tough textures and lighting issues in colonoscopy images >

Lu: The real implication is that we don't need perfect ground truth depth data available everywhere to get high accuracy for complex 3d reconstruction > <ref:2511.22250#pg1>

Meng: I’m still thinking about deployment; if this adaptation process is manageable, it opens up a path for putting these geometric models into actual clinical diagnostic workflows >

Lalam: I’m sure of what they achieved because the results on SimColthree dee were very strong, with point map accuracy around one point seven four two >

Jane: That shows that this technique works across synthetic and real datasets, which is a big deal for generalizability in medical AI >

Tom: And their performance on camera pose estimation was also competitive, with an absolute translation error of zero point zero zero six two on SimColthree dee >

Lu: The authors mentioned they plan to integrate this adapted geometric foundation model into a SLAM framework for more efficient reconstruction down the road >

Meng: That would be huge if that integration actually works smoothly at scale without needing massive retraining cycles >

Zhiyi Jiang, Yifu Wang, §uelian Cheng, Zongyuan Ge

School of Computer Science and Engineering, Southeast University · Vetex Lab, Shanghai, China · §Southeast University - Monash University Joint Graduate School, Suzhou Research Institute, Monash University · Faculty of Information Technology, Monash University

eess.IV, cs.CV

Submitted: 2025-11-27

Updated: 2026-10-21

DOI: 10.1109/LRA.2025.3623938

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist: ColonAdapter presents a self-supervised fine-tuning framework that adapts 3D geometric foundation models for colonoscopy geometry estimation by leveraging pretrained geometric priors while

Key concepts

Geometric Foundation Models: These are large AI models pre-trained on vast 3D data that understand the general geometry of scenes. ColonAdapter takes these powerful models and fine-tunes them specifically for colonoscopy images, making them better at understanding the unique shapes and challenges found in clinical settings like moving lights and textureless surfaces.
Detail Restoration Module (DRM): This is an added component that fuses high-resolution, fine details extracted by a separate convolutional encoder directly into the main foundation model. This fusion helps the model recover subtle, small features that might be lost when dealing with complex or textureless areas in colonoscopy images, leading to more accurate reconstructions.
Geometry Consistency Loss: This loss term ensures that the 3D geometric predictions made across different frames are internally consistent. It uses components like Tflow and Tpose to align the scale of estimated camera poses with the scale of predicted point maps, which prevents errors from accumulating over a video sequence.

Terminology

Foundation Model — 3× in this paper · explained in 3 episodes
Consistency — 2× in this paper · explained in 3 episodes
Foundation Models — 2× in this paper · explained in 7 episodes
Adaptation — 1× in this paper · explained in 1 episode
Fine-Tuning — 1× in this paper · explained in 5 episodes

Summary

The gist: ColonAdapter presents a self-supervised fine-tuning framework that adapts 3D geometric foundation models for colonoscopy geometry estimation by leveraging pretrained geometric priors while tailoring them to clinical data.

Problem and Motivation

Estimating 3D geometry from monocular colonoscopy images is challenging due to non-Lambertian surfaces, moving light sources, and large textureless regions (Page 1). While recent 3D geometric foundation models eliminate the need for multi-stage pipelines, their performance deteriorates in clinical scenes because they are primarily trained on natural scene datasets and struggle with specularity and homogeneous textures typical in colonoscopy, leading to inaccurate geometry estimation (Page 1). Traditional feature-based reconstruction methods struggle because of sparse and unevenly distributed key points in endoscopic images, resulting in poor reconstructions (Page 1). Existing methods often assume known ground-truth intrinsic parameters, which are often unavailable or vary across devices, and their reliance on multi-stage pipelines makes them prone to noise and error propagation (Page 1).

Proposed Framework

ColonAdapter proposes a self-supervised fine-tuning strategy using only monocular videos, without camera information or groundtruth depth (Page 2). The core idea is to demonstrate that photometric loss supervision suffices by leveraging the geometric priors of the foundation model (Page 2). To improve fine-detail recovery, the method introduces a Detail Restoration Module (DRM) that fuses fine details from an auxiliary convolutional encoder into the foundation model (Page 2).

Key Technical Contributions

The main contributions of ColonAdapter are:

Development of a self-supervised fine-tuning framework that adapts 3D geometric foundation models to colonoscopy scene through the supervision of photometric constraints (Page 2). This includes integrating the confidence map into the photometric loss for training stability enhancement, and a geometric consistency loss is introduced to ensure coherent geometric predictions across frames (Page 2).
Design of a Detail Restoration Module for ViT-based geometric foundation models, featuring an auxiliary CNN encoder and a feature fusion adapter for enhanced multi-level detail extraction and reconstruction accuracy in colonoscopy scenes (Page 2).
Comprehensive evaluation on three colonoscopy datasets, demonstrating strong performance across multiple 3D vision tasks, including camera pose estimation, monocular depth estimation, and dense point map estimation (Page 2).

Training Objectives

The training pipeline involves several components:

Adapted Foundation Module: This module takes two input images and predict[s] corresponding point maps along with their associated confidence maps (Page 3). Intrinsic parameters are estimated from the predicted point map using the Weiszfeld algorithm to minimize reprojection error (Page 3).
Affiliation Module: Used exclusively during training, this module provides image brightness calibration, optical flow, and affiliated pose information (Page 3). It uses an optical flow network to warp frames and an appearance network to calibrate brightness (Page 3).
Image Reconstruction and Loss Calculation: The target image is reconstructed by warping the source image using the predicted point map and relative pose (Page 4). The primary objective is minimizing the difference between reconstructed and target images using a photometric loss defined as: l t photo(i) = α · 1 − SSIM(˜I t(i), ˆI t(i)) squared + (1 − α) · ˜I t(i) − ˆI t(i) / (Page 4).
Confidence-Weighted Photometric Loss: This loss is defined as Lconf-photo = λconf · (1/Nt X Nt i=1 Ct i · l t photo(i) − β · log(Ct i)) + λphoto · (1/Nt−1 N Xt−1 i=1 l t−1 photo(i)) (Page 5).
Geometry Consistency Loss: This loss addresses scale inconsistencies and lack of direct supervision by having two components: Tflow, which aligns point map scales using optical flows, and Tpose, which aligns estimated poses with point map scales (Page 5). The final loss is Lgeo = λflow · Tflow + λpose · Tpose (Page 5).

Experimental Results

ColonAdapter was evaluated on three datasets: SimCol3D (synthetic), Simulated Colonscopy Dataset (CSD), and EndoMapper (real-world) (Page 4).

Camera Pose Estimation: On the SimCol3D dataset, the method achieved competitive performance compared to other methods, with its Absolute Translation Error (ATE) being 0.0062 in Table II, which is lower than AF-SfMLearner's ATE of 0.0067 (Page 6).
Monocular Depth Estimation: On SimCol3D and CSD datasets, the method achieved superior accuracy across most metrics compared to methods like DUSt3R (Page 6). For instance, in Table III, Ours achieved an RMSE of 0.183 for SimCol3D depth estimation while DUSt3R had an RMSE of 0.270 (Page 6).
Point Map Estimation: On SimCol3D, the method outperformed other methods in point map estimation accuracy even without access to ground-truth intrinsic parameters (Page 7). In Table V, Ours achieved an Accuracy of 1.742 and RMSE log of 0.111 for SimCol3D point map estimation while DUSt3R had an Accuracy of 41.535 and RMSE log of 0.486 (Page 7).

A qualitative comparison on the EndoMapper dataset showed improvements over DUSt3R, especially in challenging conditions like specularity and textureless regions (Page 7).

Ablation Study Findings

The ablation studies confirmed the effectiveness of the proposed components:

Feature Fusion: A lightweight sequential fusion strategy yielded the best performance, while more complex attention-based modules did not yield improvements (Page 8).
Impact of Loss Terms and DRM: The introduction of both loss terms and DRM continued to improve performance; notably, removing the confidence-weighted photometric loss led to performance degradation, even below the baseline (Page 8).

The framework successfully adapts foundation models for colonoscopy by introducing specific modules and losses tailored to clinical challenges. This approach shows strong performance in pose estimation, depth prediction, and point map reconstruction across synthetic and real datasets. The future work includes integrating the adapted geometric foundation model into a SLAM framework for efficient reconstruction (Page 8).

reference:2511.

Improvements for AI systems

Adapt geometric foundation models for colonoscopy geometry estimation by using a self-supervised fine-tuning strategy that leverages geometric priors while tailoring them to clinical data. This allows AI systems to accurately reconstruct scenes with large textureless areas, complex anatomical structures, moving light sources, and non-Lambertian surfaces, which current models fail to handle.
Integrate a Detail Restoration Module (DRM) into ViT-based geometric foundation models by fusing fine details from an auxiliary convolutional encoder into the model. This enables the system to achieve enhanced multi-level detail extraction and reconstruction accuracy in colonoscopy scenes.
Implement a confidence-weighted photometric loss to enhance training stability and focus on reliable regions, as described by combining the photometric loss of frame t − 1 with the confidence-aware loss of frame t. This improves robustness by ensuring that low-confidence regions in frame t are supervised when warped to frame t − 1 for photometric loss computation.
Introduce a geometry consistency loss to ensure coherent geometric predictions across frames, defined by terms like Tflow and Tpose. This addresses scale inconsistencies and lack of direct supervision for point maps, aligning the predicted camera poses scale from the affiliation module with the point maps scale.
Develop a comprehensive evaluation pipeline that tests performance across multiple 3D vision tasks, including camera pose estimation, monocular depth prediction, and dense 3D point map reconstruction, using both synthetic and real colonoscopy datasets. This provides a robust measure of the system's capability in clinical settings without requiring ground-truth intrinsic parameters.

Abstract

Estimating 3D geometry from monocular colonoscopy images is challenging due to non-Lambertian surfaces, moving light sources, and large textureless regions. While recent 3D geometric foundation models eliminate the need for multi-stage pipelines, their performance deteriorates in clinical scenes. These models are primarily trained on natural scene datasets and struggle with specularity and homogeneous textures typical in colonoscopy, leading to inaccurate geometry estimation. In this paper, we present ColonAdapter, a self-supervised fine-tuning framework that adapts geometric foundation models for colonoscopy geometry estimation. Our method leverages pretrained geometric priors while tailoring them to clinical data. To improve performance in low-texture regions and ensure scale consistency, we introduce a Detail Restoration Module (DRM) and a geometry consistency loss. Furthermore, a confidence-weighted photometric loss enhances training stability in clinical environments. Experiments on both synthetic and real datasets demonstrate that our approach achieves state-of-the-art performance in camera pose estimation, monocular depth prediction, and dense 3D point map reconstruction, without requiring ground-truth intrinsic parameters.