[Inovis: Instant Novel-View Synthesis]

Inovis: Instant Novel-View Synthesis

Mathias Harrer1,  Linus Franke1,  Laura Fink1,2,  Marc Stamminger1,  Tim Weyrich1

1 Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
2 Fraunhofer IIS, Germany


Novel-view synthesis is an ill-posed problem in that it requires inference of previously unseen information. Recently, reviving the traditional field of image-based rendering, neural methods proved particularly suitable for this interpolation/extrapolation task; however, they often require a-priori scene-completeness or costly preprocessing steps and generally suffer from long (scene-specific) training times. Our work draws from recent progress in neural spatio-temporal supersampling to enhance a state-of-the-art neural renderer's ability to infer novel-view information at inference time. We adapt a supersampling architecture [Xiao et al. 2020], which resamples previously rendered frames, to instead recombine nearby camera images in a multi-view dataset. These input frames are warped into a joint target frame, guided by the most recent (point-based) scene representation, followed by neural interpolation. The resulting architecture gains sufficient robustness to significantly improve transferability to previously unseen datasets. In particular, this enables novel applications for neural rendering where dynamically streamed content is directly incorporated in a (neural) image-based reconstruction of a scene. As we will show, our method reaches state-of-the-art performance when compared to previous works that rely on static and sufficiently densely sampled scenes; in addition, we demonstrate our system's particular suitability for dynamically streamed content, where our approach is able to produce high-fidelity novel-view synthesis even with significantly fewer available frames than competing neural methods.

Citation Style:    Publication

Inovis: Instant Novel-View Synthesis.
Mathias Harrer, Linus Franke, Laura Fink, Marc Stamminger, Tim Weyrich.
SIGGRAPH Asia 2023 Conference Papers (to appear).
Mathias Harrer, Linus Franke, Laura Fink, Marc Stamminger, and Tim Weyrich. Inovis: Instant novel-view synthesis. In SIGGRAPH Asia Conference Papers, New York, NY, USA, December 2023. Association for Computing Machinery.Harrer, M., Franke, L., Fink, L., Stamminger, M., and Weyrich, T. 2023. Inovis: Instant novel-view synthesis. In SIGGRAPH Asia Conference Papers, Association for Computing Machinery, New York, NY, USA.M. Harrer, L. Franke, L. Fink, M. Stamminger, and T. Weyrich, “Inovis: Instant novel-view synthesis,” in SIGGRAPH Asia Conference Papers. New York, NY, USA: Association for Computing Machinery, Dec. 2023. [Online]. Available: https://doi.org/10.1145/3610548.3618216


The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) under the NHR project b162dc.NHR funding is provided by federal and Bavarian state authorities. NHR@FAU hardware is partially funded by the German Research Foundation (DFG) - 440719683. Linus Franke was supported by the Bayerische Forschungsstiftung (Bavarian Research Foundation) AZ-1422-20. We thank NavVis GmbH for providing the office dataset.

Privacy: This page is free of cookies or any means of data collection. Copyright disclaimer: The documents contained in these pages are included to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.