BA-T: An Iterative Transformer for Two-View Bundle Adjustment

The gist The proposed BA-T, an iterative Transformer for two-view bundle adjustment, introduces a compact, reusable BA-T layer in latent space that implements BA-style structured updates to progressively refine camera and local geometry tokens.

How it works

BA-T is inspired by classical bundle adjustment (BA), which is viewed as an iterative information propagation process between poses and local geometry (3). It replaces deep cross-view attention stacks with a single structured iterative refinement layer, enabling more compact and efficient updates of camera poses and 3D geometry than vanilla crossattention blocks (1). The core idea is to model the information flow of BA through a learnable mechanism that emulates the interaction pattern suggested by classical BA (3.2).

The process involves three functional roles:

1. Cross-view correspondence estimation and latent residual computation (3.3).

2. Camera update, which is performed by attending from a single camera token to multiple residual-related tokens (6).

3. Geometry update, which uses crossattention between the two views to encourage consistent geometric refinement across views (9).

Formally, BA-T refines camera and geometry tokens iteratively: T(k) = n c(k)a→b, g(k)a, g(k)b o delta geo token of delta geo token of Local Geo Update Local Geometry Update A B Space TF Space TF delta cam token latent residual geo tokens in space of Latent Residual Calculation Property & Residual Function Camera-Cond Geometry Transform Correspondence Matching Camera Update Camera Update Local Geometry Update

The update process is defined by:

(4) T(k+1) = T(k) + delta T(k), where the update process can be applied iteratively for a variable number of steps k = 0,..., K, where K is not fixed and may differ between training and inference (3.2).

Key Components of BA-T

The implementation relies on several components inspired by the general form of BA:

1. Camera-Conditioned Geometry Transform: This uses an adaLN layer to condition the local geometry token g p a on the camera token ca→b, resulting in π a→b c(g p a):= Condition(g p a, ca→b) (3.3).

2. Token-level Correspondence Matching: This employs an attention-inspired correspondence method where cross-attention queries corresponding tokens in view b to obtain the attention score αp, and the corresponding token g q b is obtained as the attention-weighted sum of all geometry tokens in view b (5).

3. Latent Property and Residual Function: The residual r(·, ·) is calculated as an MLP applied to the difference between the properties of two tokens: ra→b f(g q b) − f(π a→b c(g p a)) = MLP (7).

Training and Iterative Supervision

The model is trained on point-level geometry, relative pose, and cross-view consistency constraints, with the overall training objective defined as L = λpmapLpmap + λposeLpose + λgcLgc (10). To encourage progressive refinement, supervision is applied at the output of each iteration: Ltotal = X K k=1 α K−k λ L(k), where 0 < αλ < 1 (11).

Experimental Results and Efficiency

Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieving stronger cross-view consistency than conventional decoders (2). Table 4 shows that BA-T achieves the best overall pose estimation among two-view models, even compared to models trained on multi-view data, while using only 16% of their size (4.1). In terms of iterative refinement, BA-T consistently outperforms ViSTA† w/ iterative training at all iterations (8). Furthermore, BA-T is substantially more efficient than classical solvers like Droid-SLAM, achieving approximately 5× faster inference while reducing memory consumption by about 45% (13). The extension to multi-view reconstruction is achieved by defining a view graph G(V, E) and allowing each view to attend to geometry tokens from all neighboring views in the graph (16).

The paper concludes that BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture (Abstract). The full BA-T model achieves the strongest results on both camera pose and 3D reconstruction metrics (Table 3). The final extension allows the refinement layer to be reused without modifying the network structure, meaning the multiview capability emerges purely from the changed token connectivity defined by the input view-graph (16). This demonstrates that structured iterative refinement can replace depth-heavy attention while remaining lightweight and accurate (Abstract).

--- Page 15 ---

view view view view view

Figure 9 Visualization of camera-conditioned geometry transformation. Given two input views, a and b, we visualize the point map regressed from the geometry tokens ga of view a as colored point clouds in the coordinate frame of view a. We also visualize the point map regressed from the transformed tokens π a→b c(ga), which is obtained by conditioning ga on the camera token ca→b, as blue point clouds in the coordinate frame of view b. The two point clouds align closely under the pose transformation encoded in the camera token, demonstrating that the Camera-Conditioned Geometry Transform component of BA-T operates as intended by transforming latent local geometry into the coordinate frames of other views (15).

--- Page 16 ---

Input Images Droid-SLAM BA-T Iter: 20 Total time: 160ms Iter: 4 Total time: 30ms

Figure 8 Reconstruction comparison. Both methods are initialized from BA-T (iter = 0). DroidSLAM uses ground-truth intrinsics, whereas BA-T does not. BA-T achieves stronger performance with fewer iterations and faster runtime, benefiting from its more expressive latent space (14).

--- Page 17 ---

traj ATE: 0.013m traj ATE: 0.010m traj ATE: 0.008m iteration = 1 iteration = 2 iteration = 3

Figure 6 (left) Qualitative reconstruction results. We visualize reconstructed geometry and 3D points (in local frame) error maps at iteration 1 and iteration 4. Red boxes highlight noticeable misalignments in the 1st iteration, which are corrected in the 4th iteration, as indicated by green boxes. This demonstrates the effectiveness of the iterative refinement in BA-T for improving local geometric consistency (8).

--- Page 9 ---

Table 3 Ablation study of BA-T. Removing any component degrades pose and geometry performance, emphasizing each module’s contribution (9).

--- Page 14 ---

Table 5 Model size and efficiency comparison under two-view inputs. Decoder size is reported in millions of parameters (M), peak GPU memory in GB, and running time in milliseconds (ms). BA-T achieves the most compact architecture and the highest computational efficiency among all compared methods (14).

--- Page 13 ---

Table 4 Quantitative comparison of two-view inputs on 7Scenes [30]. Method Rot AUC @5◦ Trans AUC @0.05m Rel. δ < 1.25 Time (ms) Peak Mem (GB) Droid-SLAM [35] BA-T

--- Page 8 ---

Table 2 Performance comparison on 7Scenes [30] across iterative refinement steps. Increasing the number of refinement iterations consistently improves pose and geometry accuracy (8).

--- Page 10 ---

Figure 5 (right) Iterative refinement behavior. BA-T (green curve) consistently outperforms ViSTA† w/ iterative training (red curve) and converges within 3 ∼ 4 refinement steps (8).

--- Page 12 ---

Table 6 Quantitative comparison of 4 view inputs on BundleFusion [8]. BA-T achieves comparable accuracy to prior methods while requiring significantly lower memory and substantially faster inference (16).

--- Page 17 ---

Figure 10 More qualitative 4-view results.

--- Page 5 ---

Abstract Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency (Abstract).

--- Page 2 ---

1 Introduction Estimating dense 3D structure and camera poses from multi-view images is fundamental in 3D computer vision, with applications in robotics, AR/VR, and autonomous driving. To obtain dense reconstructions, recent feed-forward methods [42, 18, 39, 17, 43, 19] directly regress dense 3D pointmaps (and camera poses) in a single forward pass. To enforce multi-view consistency, they use deep cross-view attention blocks as decoders to exchange information across images (2).

--- Page 2 ---

In contrast, classical bundle adjustment (BA) [37, 1, 2] refines camera poses and scene structure through iterative optimization, explicitly enforcing geometric consistency. However, its explicit geometric formulation makes it hard to embed in learning-based methods using implicit token representations (2).

--- Page 3 ---

Recently there are learning-based 3D reconstruction systems [33, 35, 36, 40] trying to incorporate differentiable BA modules into their pipelines to improve reconstruction accuracy by adjusting predicted feature correspondences or optical flow.

1. The system can perform structured iterative refinement of camera poses and 3D geometry by mimicking classical bundle adjustment (BA) through a single lightweight layer rather than deep cross-view attention stacks. This enables a compact, efficient, and structural alternative to depth-heavy attention, allowing for accurate 3D reconstruction within a lightweight architecture.

2. The AI system can achieve superior multi-view consistency by using the learned structure of BA: BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders. This is accomplished through the iterative update process defined by equations (4) and (9), which progressively enforces global geometric consistency across views.

3. The system can operate in a multi-view setting by generalizing the two-view formulation to use a view graph, allowing each view to attend to all other views and aggregate constraints, which effectively changes the interaction from a one-view-to-one-view cross-attention to a one-view-to-many-views cross-attention.

4. The system can be trained for progressive refinement by utilizing iterative supervision: later iterations are given higher weights, while earlier iterations contribute less, ensuring the network learn[s] to gradually refine predictions rather than only focusing on the final output.

5. The AI system can achieve high computational efficiency by replacing explicit solvers with latent updates: BA-T benefits from expressive latent representations and learned priors from training data, leading to improved robustness and accuracy, while being substantially more efficient, achieving approximately 5× faster inference while reducing memory consumption by about 45%.

Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture. Github repository: https://github.com/zhangganlin/BA-T.