ZipSplat: Fewer Gaussians, Better Splats

arXiv:2606.05102 · cs.CV · Submitted 2026-06-03 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ZipSplat: Fewer Gaussians, Better Splats".

Jane: The gist ZipSplat introduces a token-based feed-forward model that decouples Gaussian placement from the pixel grid, achieving state-of-the-art,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The paper we're looking at today is titled "ZipSplat: Fewer Gaussians, Better Splats," and it’s written by Alexander Veicht, Sunghwan Hong, Dániel Baráth, and Marc Pollefeys.

Jane: That’s a solid team of authors involved in this work. What does the title actually suggest about what they are trying to achieve?

Lu: The title itself is pretty direct: fewer Gaussians, better splats. It immediately tells you that their main goal is efficiency without sacrificing image quality.

Tom: Right, because if you’re looking at three dee Gaussian Splatting, the number of Gaussians directly dictates how much detail the scene can hold, so reducing that number while improving quality is a major win <ref:2606.05102#pg1>.

Jane: So it’s not just a small tweak; they are aiming to fundamentally change how we think about representing three dee scenes using this specific method <ref:2606.05102#pg1>.

Lu: They are trying to break away from the traditional way of thinking where the representation budget is tied directly to the input image resolution, which is what they call pixel-Gaussian coupling.

Meng: So if you’re a developer looking at this, what’s the immediate implication? Are we looking at a way to drastically cut down on model size or processing time right away?

Tom: It's both. They propose a feed-forward three dee Gaussian Splatting architecture that reconstructs scenes from posed or pose-free images in a single forward pass, which is inherently fast <ref:2606.05102#pg1,from posed or pose-free images in a single forward pass>.

Jane: So the speed comes from that single forward pass idea, but the efficiency gain comes from how they manage the representation capacity across different scene elements.

Lu: The core idea they introduce is treating scenes as compact sets of scene tokens instead of a grid of pixels, which allows them to adapt their capacity to the actual content of what’s in the scene.

Meng: That sounds promising if we can actually get those tokens clustered well enough to capture the essential geometry without losing too much detail.

Tom: They achieve this by proposing a token compression mechanism via feature-space clustering, which groups similar tokens together based on geometric and semantic similarity.

Jane: Feature space clustering sounds like a sophisticated way to find patterns in the data that aren't immediately obvious from just looking at individual pixel placements.

Lu: And they use those compressed tokens as the basis for cross- and self-attention layers, which refine them by attending back to the full visual tokens, which is how they get rich information.

Meng: So it’s a two-step process: first compress into tokens, then refine those tokens with attention. That’s an interesting pipeline to have to manage.

Tom: And finally, they use a lightweight multi-layer perceptron to decode each token into a group of Gaussians with unconstrained three dee positions, which is where the actual splatting happens <ref:2606.05102#pg1>.

Jane: So it’s moving away from a pixel-to-Gaussian direct prediction and towards this more abstract token-based representation, which sounds much more flexible.

The paper's summary: Tom: We talked about the architecture, but what is the core problem they are solving in plain terms? It boils down to the inefficiency of per-pixel formulation.

Jane: Okay, when you’re rendering scenes as sets of Gaussians, current feed-forward methods predict one Gaussian for every pixel, and that ties your total representation budget directly to how high resolution your input images are.

Lu: That’s the problem: a flat wall gets the same capacity as a richly textured object just because they occupy the same number of pixels.

Meng: So you’re wasting resources on redundant information, which is exactly what we see happening in large-scale scene understanding tasks.

Tom: And overlapping views make things even worse because they produce duplicate Gaussians for the same surface, causing memory to grow linearly with every extra input image without any proportional gain in quality.

Jane: So the naive question a listener would ask is, why do we keep doing this if it’s so inefficient?

Lu: ZipSplat sidesteps this by predicting compact sets of scene tokens directly instead of from pixels, which allows the capacity to adapt to scene content.

Tom: The token compression mechanism via feature-space clustering groups similar tokens together by geometric and semantic similarity, effectively merging those redundant observations from different views.

Jane: So when you compress down into K scene tokens at inference time, you’re essentially grouping all the information about a specific surface together into one manageable piece of data.

Lu: And they say this compression ratio r is adjustable at inference, giving a single trained model a continuous knob to adjust the Gaussian budget without needing to retrain it.

Meng: That means we can switch between high-fidelity reconstruction and something much faster and smaller just by picking a different compression setting during the inference pass.

Tom: And they achieve this while maintaining state-of-the-art performance on DLthree deeV and RealEstate10K while using up to thirty-three times fewer Gaussians at comparable quality <ref:2606.05102#pg1,fewer Gaussians at comparable quality>.

Jane: So, the paper’s main summary is that they’ve created a way to predict Gaussians from sparse images in one pass that doesn't tie the representation budget to camera resolution.

The paper's improvements: Tom: Let’s get into the specific improvements they propose, because those are the concrete things we need to understand how this actually helps.

Lu: One major improvement is exactly what we discussed: decoupling Gaussian placement from the 2D pixel grid, which fundamentally changes how capacity is managed <ref:2606.05102#pg1>.

Jane: And they pair that with a continuous knob for inference-time budget control through their token compression mechanism, which means you can trade speed for detail on the fly.

Lu: They also introduce an explicit geometric supervision loss against ground-truth three dee points P back-projected from depth maps to pull unconstrained Gaussians toward valid scene surfaces <ref:2606.05102#pg1>.

Meng: That geometric supervision is a big step because it restores the implicit grounding that was lost by removing ray-anchoring during training, which is something I’ve been thinking about when designing systems.

Tom: And they define their total objective as a combination of losses: an L1 photometric loss, an Lrgb loss, an LPIPS perceptual loss, and that geometric supervision loss against ground-truth three dee points P back-projected from depth maps <ref:2606.05102#pg1>.

Jane: So it’s not just one thing; they are balancing image fidelity with geometry to ensure the output looks realistic and matches the input structure.

Lu: They also include an L1 depth loss as part of that training objective, which helps stabilize things early on by pulling the depth estimation closer to reality.

Meng: From a production standpoint, stabilizing training this way is crucial because we don't want a model that just breaks when it encounters slightly different input data in production.

Tom: They also start with a coupled initialization where all Gaussians of a token have identical parameters and low initial opacity to prevent early occlusions.

Jane: And the progressive schedules they use to grow the number of context views from two up to twenty-four, which helps show how the system handles increasing context views gracefully.

Lu: The key is that they also show that ZipSplat generalizes zero-shot capability to Mip-NeRF360 and ScanNet++, scaling gracefully as context views grow where pixel-aligned methods degrade.

Meng: So it’s not just about making it work for the specific datasets they tested; it’s about showing the architecture is robust enough to handle more complex three dee scenes naturally <ref:2606.05102#pg1>.

Tom: And to push things further, they also mentioned a test-time optimization where freezing the decoder and optimizing just the scene tokens at a fixed Gaussian budget can add about five decibels in PSNR <ref:2606.05102#pg2>.

Jane: That’s an interesting finding; it means even refining the token representations alone offers substantial quality gains for a given representation size.

Conclusion: Tom: So, to wrap up "ZipSplat: Fewer Gaussians, Better Splats," this architecture successfully decouples primitive placement from the 2D pixel grid by predicting unconstrained Gaussians from compact scene tokens <ref:2606.05102#pg1,ZipSplat: Fewer Gaussians, Better Splats>.

Jane: The main result is achieving state-of-the-art pose-free novel view synthesis on DLthree deeV and RealEstate10K while using up to six times fewer Gaussians than pixel-aligned methods <ref:2606.05102#pg1>.

Lu: It’s a model that offers continuous control over the Gaussian budget through their token compression mechanism, allowing you to adjust the budget without retraining.

Meng: For deployment, that knob is valuable because it lets us choose the right trade-off between speed and detail for any given scene complexity during inference.

Tom: And they show that this system maintains sharp, detailed reconstructions as input coverage increases, unlike other methods that blur out.

Jane: So the implication is that we can build three dee scenes with significantly fewer primitives while maintaining high fidelity even when we’re using more input data <ref:2606.05102#pg1>.

Lu: Looking ahead, ZipSplat’s compact representation could become really useful for semantic scene understanding because those per-Gaussian features might quickly dominate memory usage.

Meng: And the authors also suggest extending this concept to 4D dynamic scenes where token clustering could compress redundant observations across both space and time.

Tom: It's a solid piece of work that proves that representation learning can unlock significant efficiency gains in three dee reconstruction tasks <ref:2606.05102#pg1>.

Jane: It’s a paper that gives us a concrete method for managing the trade-off between speed and detail by making it adjustable at inference time.

ETH Zurich · Microsoft

cs.CV

Submitted: 2026-06-03

Updated: 2026-10-19

Project page: https://veichta.com/zipsplat

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: The gist ZipSplat introduces a token-based feed-forward model that decouples Gaussian placement from the pixel grid, achieving state-of-the-art, pose-free novel view synthesis while using up to 6×

Key concepts

3D Gaussian Splatting (3DGS): A method that represents 3D scenes as a collection of anisotropic 3D Gaussians. These Gaussians are rendered using differentiable rasterization, which allows for fast and explicit high-quality view synthesis of the scene.
Scene Tokens: Instead of processing every pixel, ZipSplat treats scenes as compact sets of tokens. A foundation model extracts dense visual tokens from images, which are then compressed via k-means clustering into a smaller set of scene tokens that capture the essential geometric and semantic information.
Decoupling Placement and Compression: ZipSplat separates where Gaussians are placed from how many there are. It predicts compact scene tokens first, which group similar observations. This token compression merges redundant surface observations, meaning the model can manage memory efficiently by grouping similar geometry together.
Geometric Supervision Loss (Lgeo): This is an explicit loss function added to training that pulls the unconstrained Gaussians toward valid scene surfaces derived from ground-truth depth maps. This supervision restores the geometric grounding that is typically provided by pixel alignment, ensuring the reconstructed Gaussians adhere to real-world scene geometry.

Terminology

Clustering — 2× in this paper · explained in 2 episodes
3D Gaussian Splatting — 1× in this paper · explained in 2 episodes
Decode — 1× in this paper · explained in 2 episodes
Gaussian Splatting — 1× in this paper · explained in 4 episodes
K-means Clustering — 1× in this paper · explained in 3 episodes

Summary

The gist ZipSplat introduces a token-based feed-forward model that decouples Gaussian placement from the pixel grid, achieving state-of-the-art, pose-free novel view synthesis while using up to 6× fewer Gaussians than pixel-aligned baselines.

Introduction and Motivation

3D Gaussian Splatting (3DGS) represents scenes as sets of anisotropic 3D Gaussians rendered via differentiable rasterization, enabling fast, explicit, and high-quality view synthesis The standard per-scene optimization requires minutes to hours and dense multi-view input Feed-forward methods sidestep this cost by predicting Gaussians from sparse images in a single network pass However, current approaches predict one Gaussian per input pixel, tying the representation budget to camera resolution rather than scene complexity This per-pixel formulation introduces inefficiencies because a flat wall and a richly textured object receive the same Gaussian capacity simply because they occupy the same number of pixels Furthermore, overlapping views produce duplicate Gaussians for the same surface, growing memory linearly with the number of input images without a proportional gain in quality

ZipSplat Architecture and Mechanism

ZipSplat is a feed-forward architecture that treats scenes not as a grid of pixels but as a compact set of scene tokens The process involves several key steps:

Multi-view foundation model extracts dense visual tokens from the N images
These visual tokens are compressed via k-means clustering in feature space into K scene tokens, with K set by a compression ratio at inference
Cross- and self-attention layers refine these scene tokens by attending back to the full visual tokens
A lightweight MLP then decodes each token into a group of Gaussians with unconstrained 3D positions

Decoupling Placement and Compression

ZipSplat achieves decoupling by predicting the compact set of scene tokens directly instead of from pixels The token compression mechanism via feature-space clustering groups similar tokens by geometric and semantic similarity, merging redundant observations of the same surface from different views The compression ratio r is adjustable at inference, giving a single trained model a continuous quality–efficiency knob without retraining

Training and Supervision

To stabilize training without the implicit grounding that pixel alignment provides, ZipSplat applies an explicit geometric supervision loss that pulls unconstrained Gaussians toward valid scene surfaces The total training objective is defined as a combination of losses: an L1 photometric loss Lrgb, an LPIPS perceptual loss Llpips, a geometric supervision loss Lgeo against ground-truth 3D points P back-projected from depth maps, and an L1 depth loss Initialization involves coupled initialization to start all G Gaussians of a token with identical parameters, low initial opacity to prevent early occlusions, and normalizing scene geometry to unit scale Progressive schedules involve growing the number of context views N from 2 to 24 and following a cosine schedule for the compression ratio r

Performance and Efficiency

ZipSplat sets a new state of the art on DL3DV and RealEstate10K with ∼6× fewer Gaussians than pixel-aligned methods, surpassing the best pose-free baseline by 2.1 dB and 1.2 dB PSNR The model maintains sharp, detailed reconstructions as input coverage increases, whereas YoNoSplat and DA3 produce blurrier renders despite using an order of magnitude more Gaussians Inference cost analysis shows that the backbone dominates forward pass time and memory at every view count, but token compression and Gaussian decoding remain negligible under scaled compression Under view-dependent scaling, memory remains nearly flat because the clustering mechanism merges redundant cross-view observations rather than linearly accumulating them ZipSplat degrades gracefully down to r=0.1, well below the compression seen during training, and k-means outperforms stride by 0.45 PSNR at r=0.1

Conclusion and Future Work

ZipSplat presents a feed-forward 3D Gaussian Splatting architecture that decouples primitive placement from the 2D pixel grid by predicting unconstrained Gaussians from a compact set of scene tokens This formulation achieves state-of-the-art, pose-free novel view synthesis while using up to 6× fewer Gaussians than pixel-aligned baselines and offering continuous inferencetime control over the Gaussian budget The work opens two paths for future research: first, ZipSplat’s compact representation benefits semantic scene understanding, where per-Gaussian features quickly dominate memory Second, decoupling primitives from the spatial grid offers a direct extension to 4D dynamic scenes, where token clustering can compress redundant observations across both space and time

References

1 An, H., Jung, J., Kim, M., Hong, S., Kim, C., Fukuda, K., Jeon, M., Han, J., Narihira, T., Ko, H., et al.: C3G: Learning compact 3D representations with 2K Gaussians. arXiv preprint arXiv:2512.04021 (2025)

4 Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: Endto-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)

8 Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: CVPR. pp. 605–613 (2017)

9 Fan, Z., Cong, W., Wen, K., Wang, K., Zhang, J., Ding, X., Xu, D., Ivanovic, B., Pavone, M., Pavlakos, G.: InstantSplat: Sparse-view Gaussian Splatting in seconds. arXiv preprint arXiv:2403.20309 (2024)

11 Hong, S., Jung, J., Shin, H., Han, J., Yang, J., Luo, C., Kim, S.: PF3plat: Pose-free feed-forward 3D Gaussian Splatting. arXiv preprint arXiv:2410.22128 (2024)

14 Jiang, L., Mao, Y., Xu, L., Lu, T., Ren, K., Jin, Y., Xu, X., Yu, M., Pang, J., Zhao, F.: AnySplat: Feed-forward 3D Gaussian Splatting from unconstrained views. ACM Transactions on Graphics (TOG) 44(6), 1–16 (2025)

19 Lin, H., Chen, S., Liew, J., Chen, D.Y., Li, Z., Shi, G., Feng, J., Kang, B.: Depth Anything 3: Recovering the visual space from any views. arXiv preprint arXiv:2511.10647 (2025)

39 Xu, H., Peng, S., Wang, F., Blum, H., Barath, D., Geiger, A., Pollefeys, M.: DepthSplat: Connecting Gaussian Splatting and depth. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 16453–16463 (2025)

41 Ye, B., Liu, S., Xu, H., Li, X.

Improvements for AI systems

The ZipSplat architecture can achieve state-of-the art pose-free novel view synthesis on DL3DV and RealEstate10K with ∼6× fewer Gaussians than pixel-aligned methods, and its quality remains stable as context views grow, where per-pixel methods degrade. This allows the system to reconstruct scenes with significantly fewer primitives while maintaining high fidelity as input data increases.
The introduction of a single trained model spans the entire quality–efficiency curve (Fig. 1), from a high-fidelity to a compact reconstruction, selected by one compression ratio set during inference enables continuous control over the Gaussian budget without retraining, allowing users to select the optimal trade-off between speed and detail for any given scene complexity.
The token compression mechanism via k-means clustering in feature space allows redundant observations of the same surface from different views to merge, which is shown to grow with input views, enabling the system to effectively manage memory scaling and maintain sharp geometry even at aggressive compression ratios like r=0.1.
The system can be improved by utilizing token test-time optimization where freezing the decoder and optimizing the scene tokens, at a fixed Gaussian budget can add about 5 dB PSNR, demonstrating that refining token representations alone offers substantial quality gains for a given representation size.
The integration of geometric supervision using the loss function Lgeo = (1/G) sum over gmean in G min over p in P gmean - p 2 ensures that unconstrained Gaussians are pulled toward valid scene surfaces, effectively restoring the implicit geometric grounding lost by removing ray-anchoring.

Abstract

Feed-forward 3D Gaussian Splatting reconstructs scenes from images in seconds. Pixel-aligned methods keep every Gaussian grounded in an observation, but produce redundant, potentially misaligned Gaussians across views. Token-based methods instead aggregate all views into a fixed set of learned tokens but lack explicit observation support and use a fixed reconstruction budget. We introduce ZipSplat, a method that predicts a compact, observation-grounded scene representation whose size is set at inference. It derives each scene token from an image patch rather than a learned query, and decodes it into freely placed 3D Gaussians. Tokens across views are merged into a compact set whose size is adjustable at inference. However, without geometric supervision, decoded Gaussians can drift from their source patch, and merging can mix tokens from different surfaces. We therefore introduce two geometric losses to ground and shape this representation. A cone loss ties each token's Gaussians to its source patch, and GeoCE aligns token similarity with 3D proximity so that tokens observing the same surface merge together. Building on these observation--geometry associations, we further introduce TokenBA, a token-level bundle adjustment that refines scene tokens and camera poses in a few seconds. ZipSplat requires neither camera poses nor intrinsics, yet outperforms pixel-aligned and token-based baselines on DL3DV and RealEstate10K, even those given ground-truth cameras. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++. Our project page is at https://veichta.com/zipsplat.