TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction

The gist The proposed framework TTSA3R alleviates catastrophic forgetting in streaming 3D reconstruction through explicit temporal-spatial adaptive updates

How it works

TTSA3R maintains a fixed-size persistent state that encodes scene geometry and produces per-frame 3D reconstructions. This framework consists of two main components: the Temporal Adaptive Update Module and the Spatial Context Update Module, which provide complementary signals for fine-grained state updates.

The Temporal Adaptive Update Module (TAUM) is designed to track temporal state changes by measuring temporal stability from state change magnitudes across consecutive frames. Specifically, it measures the state evolution magnitude for each token and normalizes the magnitude by the global average to obtain ∆ˆt. A sigmoid gating function is then applied to derive the temporal mask Mtemp = σ(∆ˆt − τ). Tokens with normalized changes above τ incorporate new information, while tokens below τ retain their historical representations.

The Spatial Context Update Module (SCUM) measures spatial interaction between states and observations by combining cross-attention alignment and feature consistency. It computes the spatial mask as Mspat = σ(maxspat(At ⊙ Dt)), where At is the cross-attention map and Dt is the feature divergence between consecutive frames. The final adaptive mask is obtained by fusing these complementary perspectives through element-wise multiplication, resulting in Mfinal = Mtemp ⊙ Mspat. This fusion ensures updates only when both temporal dynamics and spatial correspondence are met.

Key Contributions

The main contributions of TTSA3R include:

1. Proposing a novel framework TTSA3R to alleviate long-term catastrophic forgetting for online streaming 3D reconstruction.

2. Designing a Temporal Adaptive Update Module (TAUM) based on state evolution analysis to track temporal state changes, which enables preservation of stable information and adapts to dynamic change.

3. Introducing a Spatial Context Update Module (SCUM) combining cross-attention alignment and feature consistency to identify update-worthy regions, so as to prevent erroneous updates when prior observations lack spatial coverage.

Experimental Results

Extensive experiments show that TTSA3R achieves competitive performance on standard short-sequence benchmarks and provides substantially stronger robustness on extended sequences. On NRGBD, as sequences extend from 50 to 250 frames, TTSA3R exhibits only a 1.33× error increase, compared with over 4× degradation for CUT3R. The method achieves the best performance among streaming methods on KITTI [19] dataset under per-sequence scale metric depth accuracy. Furthermore, TTSA3R demonstrates superior performance on Bonn [35] dataset and approaches the performance of full-attention pipelines based on metric scale. In 3D reconstruction, our method exhibits only 1.33x error increase compared to over 4x degradation in NRGBD [2] dataset.

Efficiency and Robustness

TTSA3R matches TTT3R’s [9] efficiency at 18.0 FPS and 6 GB memory consumption without introducing extra learnable parameters or forward passes. The method maintains significantly lower pose error and surpasses the other two methods on TUM-dynamics [42] and ScanNet [14] datasets when assessing long-term stability. Qualitative visualizations reveal that CUT3R produces severely distorted geometry, whereas TTSA3R generates more coherent reconstruction results with accurate scene structure. The method's selective sparsity in the final mask matches the intended design of AND-logic fusion, where updates occur only when both temporal change and spatial relevance are present.

Conclusion

By extracting complementary temporal and spatial signals from internal decoder representations, TTSA3R alleviates catastrophic forgetting in streaming 3D reconstruction. This enables the model to maintain long-term consistency across extended sequences while adapting to new observations >. The framework demonstrates that the gains of our method stem from the adaptive update mechanism itself, not merely from limiting state retention over long horizons.

The paper is an arXiv preprint titled TTSA3R: Training-Free Temporal-Spatial Adaptive Persistent State for Streaming 3D Reconstruction, and it matters because it introduces a novel training-free framework that effectively mitigates catastrophic forgetting in streaming 3D reconstruction by combining temporal state evolution analysis with spatial observation quality for adaptive state updates.

--- Page 1 ---

The gist The proposed framework TTSA3R alleviates catastrophic forgetting in streaming 3D reconstruction through explicit temporal-spatial adaptive updates

--- Page 2 ---

The gist The proposed framework TTSA3R alleviates catastrophic forgetting in streaming 3D reconstruction through explicit temporal-spatial adaptive updates

How it works

TTSA3R maintains a fixed-size persistent state that encodes scene geometry and produces per-frame 3D reconstructions.

1. Improve long-term stability of streaming 3D reconstruction by implementing TTSA3R to mitigate catastrophic forgetting, as it alleviates this issue through temporal-spatial adaptive updates to achieve coherent 3D reconstruction with accurate camera poses. This allows the system to maintain structural integrity across extended sequences, preventing the severe geometric collapse seen in CUT3R.

2. Enhance camera pose estimation accuracy on long sequences by utilizing the Temporal Adaptive Update Module (TAUM) which tracks temporal state evolution across frames, enabling it to combat progressive drift accumulation during continuous motion. This results in significantly lower Absolute Translation Error (ATE), as demonstrated by TTSA3R achieving an ATE of 0.092 compared to CUT3R's drift.

3. Improve geometric fidelity in 3D reconstruction by employing the Spatial Context Update Module (SCUM) to localize spatial regions that require updates through observation-state alignment and scene dynamics. This ensures that updates only when both temporal dynamics and spatial correspondence are met, preserving structural integrity across both scenarios where baseline methods like CUT3R produce severely distorted geometry.

4. Develop a training-free, real-time streaming 3D reconstruction pipeline capable of processing long sequences with high efficiency (18.0 FPS and 6 GB memory consumption). This system can maintain competitive performance on extended sequences, achieving only a 1.33× error increase on NRGBD compared to over 4× degradation for CUT3R.

5. Implement an adaptive update strategy that selectively preserves stable information by using the temporal mask derived from Equation (6), where tokens with normalized changes above a threshold incorporate new information, while those below retain historical representations. This mechanism is controlled by the parameter τ, which is set to 1.5 for principled selection of tokens with above-average temporal changes across all scenes and datasets.

6. Achieve superior performance in video depth estimation by applying the final fused mask derived from Equation (10), where updates occur only when both signals are present (element-wise multiplication). This fusion strategy achieved the best performance of 0.079 Abs Rel and 96.6% accuracy on Bonn, showing that updates only when both temporal change and spatial relevance are present.

7. Enhance robustness against occlusions during reconstruction by using SCUM's signal, which measures cross-attention alignment confidence alongside feature divergence across consecutive frames. This allows the system to distinguish between active geometric refinement and stable geometry, preventing erroneous updates when prior observations lack spatial coverage.

8. Improve inference efficiency by matching the efficiency of TTT3R [9] at 18.0 FPS and 6 GB memory consumption without introducing extra learnable parameters or forward passes, achieving a competitive efficiency-memory trade-off among existing streaming methods.