Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement

arXiv:2511.18672 · cs.CV · Submitted 2025-11-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement".

Jane: Here's the gist of the paper. The gist Sphinx presents a training-free hybrid inference framework that combines regression-based fast initialization with diffusion-based selective refinement to achieve diffusion-level fidelity at significantly lower compute, establishing a new Pareto frontier between quality and latency in Novel View Synthesis.

Tom: Novel View Synthesis (NVS) is the task of generating new images of a scene from viewpoints that were not part of the original input<ref:2511.18672#pg4> NVS aims to generate unseen views of a scene from any number of input images and has recently attracted significant attention in AR/VR, 3D reconstruction, and freeviewpoint video applications<ref:2511.18672#pg2>.

Jane: Based on the model design, NVS methods can be classified into two categories: regression-based and diffusion-based<ref:2511.18672#pg4>.

Tom: Regression models predict target views directly from scene geometry in a single feed-forward pass, achieving real-time performance but often yielding oversmoothed results<ref:2511.18672#pg4>.

Paper summary: Tom: So, to wrap up this Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement paper, it’s a system that uses regression for fast initialization followed by diffusion for selective refinement to get high quality with much lower compute than pure diffusion >

Jane: It’s about finding that sweet spot between the speed of regression and the fidelity of diffusion, giving us flexible control over performance and quality based on what the scene actually requires >

Lu: The authors prove that this hybrid approach successfully addresses several key design challenges, like adaptive noise injection and ensuring temporal consistency under partial refinement >

Meng: The real implication for us is that we can start deploying NVS systems in AR or VR where latency matters, because we have a way to control the computational cost dynamically instead of just accepting the high cost >

Lalam: So it settles this critical gap between regression and diffusion by showing a training-free method to achieve diffusion-level fidelity at significantly lower compute levels >

Conclusion: Tom: So Sphinx is basically this new way to do Novel View Synthesis that uses regression first for speed and then diffusion for quality refinement, right?

Jane: Yeah, it’s a hybrid system that tries to bridge the gap between fast but blurry results and slow but sharp ones.

Lu: The authors are focusing on making this training-free. They're using a regression model to get a rough idea quickly, and then only running the expensive diffusion steps where they actually need them.

Meng: From my point of view, I’m really interested in the serving part here. How do you keep that whole process running efficiently when someone asks for a view?

Lalam: The core idea is using those regression outputs to guide the diffusion, which means we don't have to run as many denoising steps overall. It’s about making the diffusion work smarter, not harder.

Tom: So the main claim is they got this level of visual quality, like diffusion-based models usually get, but at a much faster speed and with less computation.

Jane: That’s what they’re showing—a big speedup in inference while keeping the image quality pretty good across different test sets.

Lu: They tackled some tough problems there too, like figuring out exactly how much noise to add before starting the diffusion refinement based on how good the initial regression guess is.

Meng: I worry about scaling that adaptive noise scheduling. If you have a million requests coming in at once, does that logic actually keep up without blowing the GPU resources?

Lalam: The paper shows they developed this adaptive decision logic by looking at scene complexity, so it should handle varying scenes better than just a fixed setting.

Tom: It sounds like they proved that you can tune the trade-off between speed and quality without having to retrain or fine-tune the diffusion model itself.

Jane: Exactly, it’s about giving users more control over how much computation they want to spend for each individual view they generate.

Lu: What’s really interesting is how they mapped out that Pareto frontier—showing where you can get better quality for less compute and where you can get more speed with just a little quality drop.

Meng: So it’s not just about getting a good result; it’s about having a system that scales reliably across different hardware setups, which is what I need to see in production.

Lalam: I'm pretty sure this approach sets a new standard for serving NVS because it makes the complex diffusion models practical for real-time applications without needing massive infrastructure.

Yuchen Xia, Souvik Kundu, Mosharaf Chowdhury, Nishil Talati

University of Illinois Urbana-Champaign · University of Michigan

cs.CV

Submitted: 2025-11-24

Updated: 2026-10-16

Code: https://github.com/stsxxx/Sphinx

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 84/100

The gist: The gist Sphinx presents a training-free hybrid inference framework that combines regression-based fast initialization with diffusion-based selective refinement to achieve diffusion-level fidelity at

Key concepts

Regression-based NVS: These models predict the target novel view directly from input images using a single, fast calculation. They are very quick but often produce blurry or oversmoothed results. Sphinx uses these models first to generate a rough, initial estimate of the scene view quickly.
Diffusion-based NVS: These models synthesize high-fidelity views through an iterative denoising process, which requires many computational steps. They produce much sharper and more realistic images than regression methods but are very slow. Sphinx uses this method selectively for areas needing high detail.
Selective Refinement: This technique focuses the expensive diffusion process only on specific parts of an image that require improvement, such as uncertain or degraded regions. Instead of refining the entire image every time, it targets only the necessary pixels to save computation and speed up generation.
SLO-aware Serving Framework: This is a system component that dynamically manages GPU resources based on how much work each request has left and how urgent it is. It intelligently allocates processing power to ensure fast responses while meeting strict service level objectives, optimizing performance across different scenes.

Terminology

Diffusion Model — 2× in this paper · explained in 10 episodes
Fidelity — 2× in this paper · explained in 2 episodes
Novel View Synthesis — 2× in this paper
Adaptation — 1× in this paper · explained in 1 episode
Novel View Synthesis (NVS) — 1× in this paper · explained in 2 episodes

Summary

The gist Sphinx presents a training-free hybrid inference framework that combines regression-based fast initialization with diffusion-based selective refinement to achieve diffusion-level fidelity at significantly lower compute, establishing a new Pareto frontier between quality and latency in Novel View Synthesis.

Background and Motivation

Novel View Synthesis (NVS) is the task of generating new images of a scene from viewpoints that were not part of the original input<ref:2511.18672#pg4> NVS aims to generate unseen views of a scene from any number of input images and has recently attracted significant attention in AR/VR, 3D reconstruction, and freeviewpoint video applications<ref:2511.18672#pg2>. Based on the model design, NVS methods can be classified into two categories: regression-based and diffusion-based<ref:2511.18672#pg4>. Regression models predict target views directly from scene geometry in a single feed-forward pass, achieving real-time performance but often yielding oversmoothed results<ref:2511.18672#pg4>. Diffusion-based models leverage strong generative priors to synthesize highfidelity, temporally consistent views through iterative denoising, which comes with high computational cost<ref:2511.18672#pg4>. This dichotomy embodies a fundamental trade-off between speed and visual quality<ref:2511.18672#pg4>.

Proposed Hybrid Approach

Sphinx proposes to use regressionbased fast initialization to guide and reduce the denoising workload for the diffusion model<ref:2511.18672#pg4>. The framework first uses a regression model for fast initialization to predict coarse target views, effectively reducing the number of denoising steps required in the subsequent diffusion refinement stage<ref:2511.18672#pg4>. It further integrates selective refinement with adaptive noise scheduling, allowing more compute to uncertain regions and frames<ref:2511.18672#pg4>. The approach combines the efficiency of regression models with the fidelity of diffusion models<ref:2511.18672#pg4>.

Design Challenges and Solutions

The paper addresses four key design challenges that must be addressed to fully leverage this design<ref:2511.18672#pg5>.

(Challenge 1)

Challenge 1 is the adaptive noise injection for regression outputs, which requires determining the appropriate amount of noise to add to each regression output before invoking the diffusion model<ref:2511.18672#pg5>. The design goal (DG1) is to adaptively determine the noise level for each regression output based on its predicted quality, injecting only as much noise as needed to balance refinement fidelity and efficiency<ref:2511.18672#pg5>.

(Challenge 2)

Challenge 2 is maintaining temporal consistency under partial denoising, which requires a mechanism that preserves temporal coherence even when only a subset of frames are refined<ref:2511.18672#pg5>. The design goal (DG2) is to maintain temporal coherence during partial denoising by preserving motion and structural consistency, even when only a subset of frames are refined<ref:2511.18672#pg5>.

(Challenge 3)

Challenge 3 involves efficient sparse refinement within frames, motivating a strategy where diffusion focuses computation only on uncertain, degraded, or newly revealed content rather than the entire frame<ref:2511.18672#pg5>. The design goal (DG3) is to enable effective, spatially selective refinement that concentrates computation on uncertain regions while preserving spatial correlations for coherent visual quality<ref:2511.18672#pg5>.

(Challenge 4)

Challenge 4 is adaptive decision logic for varying scene characteristics, requiring a system to offer controllable trade-offs between visual quality and inference speed across diverse scenes<ref:2511.18672#pg5>. The design goal (DG4) is to adaptively balance quality and performance by adjusting denoising steps and refinement scope based on scene complexity and viewing conditions<ref:2511.18672#pg5>.

System Design Components

Sphinx incorporates several components to realize the hybrid framework<ref:2511.18672#pg6>.

(Adaptive Noise Scheduling)

To address Challenge 1, Sphinx employs an adaptive noise scheduling strategy that balances efficiency and quality by enforcing a quality target defined as QSphinx(k) ≥ α · Qfull<ref:2511.18672#pg6>. It uses a non-reference image quality metric, MUSIQ, to obtain a quality score Qbreg for each regression output<ref:2511.18672#pg6>. The adaptive k-decision logic determines the starting denoising step based on the ratio between the regression model’s MUSIQ score and the interpolated reference quality<ref:2511.18672#pg6>.

(Temporally Consistent Partial Denoising)

To address Challenge 2, Sphinx introduces a latent cache mechanism that periodically caches intermediate features from attention layers to ensure spatial and temporal consistency while avoiding redundant computation<ref:2511.18672#pg6>. This allows frames not being refined to retrieve and reuse cached latents for partial denoising steps<ref:2511.18672#pg6>.

(Spatially Selective Refinement)

To address Challenge 3, Sphinx combines two complementary cues for region selection: opacity-based confidence, derived from the regression model's per-pixel opacity map, and blur-based uncertainty, derived from a Laplacian-based blur detection algorithm<ref:2511.18672#pg6>. The resulting unified spatial refinement mask is used to forward only pixels within this mask to the diffusion model for refinement<ref:2511.18672#pg6>.

(Adaptive Performance-Quality Control)

To address Challenge 4, Sphinx extends empirical analysis to derive adaptive k-decision logic across different scene types and datasets by computing CLIP embeddings of input views and performing clustering<ref:2511.18672#pg6>. This allows simple scenes to use shallower refinement while complex scenes invoke deeper refinement, enabling fine-grained regulation of runtime cost without manual tuning<ref:2511.18672#pg6>.

Evaluation and Results

Sphinx is evaluated on three diverse datasets: RE10K (Zhou et al., 2018), DL3DV (Ling et al., 2024) and ACID (Liu et al., 2021)<ref:2511.18672#pg7>. The evaluation uses metrics such as MUSIQ, PSNR, SSIM, and LPIPS<ref:2511.18672#pg10>. Across all datasets, Sphinx achieves up to 2.7× speedup over full diffusion inference while maintaining comparable visual quality<ref:2511.18672#pg7>. Specifically, the system achieves up to 2.2× speedup with less than 5% degradation in perceptual quality<ref:2511.18672#pg7>. The results validate that Sphinx resides on the Pareto frontier, demonstrating its ability to achieve an optimal balance between fidelity and efficiency<ref:2511.18672#pg7>.

Conclusion

Sphinx provides flexible control over the performance-quality trade-off through tunable runtime parameters, allowing users to adapt generation quality and computational cost on demand<ref:2511.18672#pg9>. The paper concludes that Sphinx achieves an average 1.8× (up to 2.2×) speedup over full diffusion inference with less than 5% perceptual degradation, demonstrating effective and adaptive control of the quality–performance trade-off<ref:2511.18672#pg10>. The hybrid design enables flexible navigation along the quality–latency trade-off curve, allowing the system to balance perceptual fidelity and efficiency according to runtime constraints<ref:2511.18672#pg9>. The paper establishes a new Pareto frontier for NVS without needing model training<ref:2511.18672#pg10>.

References

Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation<ref:2511.18672#pg15>.

Berian, A. and Mahalanobis, A. Modern novel view synthesis algorithms: a survey<ref:2511.18672#pg4>.

Cai, J. and Lu, H. Nerf-based multi-view synthesis techniques: A survey<ref:2511.18672#pg5>.

Charatan, D., Li, S. L.

Improvements for AI systems

Adaptive noise scheduling based on regression output quality allows Sphinx to dynamically select denoising starting points, balancing refinement fidelity and efficiency by injecting only as much noise as needed to balance refinement fidelity and efficiency.
Latent cache mechanism preserves temporal consistency during partial denoising by caching intermediate features from attention layers, ensuring temporal attention to maintain full temporal context without recomputing features.
Spatially selective refinement restricts diffusion computation only to uncertain regions by combining opacity-based confidence and blur-based uncertainty masks, which allows the system to concentrate computation on uncertain regions while preserving spatial correlations for coherent visual quality.
Scene-aware adaptation dynamically adjusts denoising steps based on input image embeddings clustered using a CLIP model, enabling fine-grained control over the quality–performance trade-off by linking visual semantics to adaptive refinement logic.
The system can perform selective refinement configurations with distinct trade-off behaviors by adjusting opacity or blur thresholds, allowing users to adapt generation quality and computational cost on demand.

Abstract

Novel View Synthesis (NVS) generates novel views of a scene from a sparse set of input images and is a key component in applications such as immersive rendering and 3D content generation. Existing NVS approaches broadly exhibit an efficiency--quality tradeoff: regression-based models generate novel views with relatively low latency but can struggle with perceptual quality, particularly in challenging regions or regions not visible in the input views (disoccluded regions), while diffusion-based models produce higher-quality and temporally consistent views at substantially greater computational cost. This motivates a hybrid approach that leverages the complementary strengths of regression and diffusion to bridge this efficiency--quality gap. We present Sphinx, a training-free hybrid inference and serving system for NVS that combines regression-level efficiency with diffusion-level quality. Sphinx first uses fast regression-based NVS to generate coarse novel views, which provide a strong initialization for diffusion and avoid expensive generation from pure noise. It then selectively applies diffusion refinement where additional computation is needed, reducing the denoising workload while preserving the quality benefits of diffusion-based NVS. To efficiently serve the resulting heterogeneous requests, Sphinx further introduces an SLO-aware serving framework that dynamically allocates GPU resources based on each request's remaining computation and urgency. Evaluated across diverse NVS models, datasets, and serving configurations, Sphinx achieves an average 2.3 times inference speedup (up to 2.7 times) with less than 10% perceptual degradation, while its SLO-aware scheduler achieves up to 10 times higher SLO attainment than FCFS scheduling across hardware platforms.