PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applications

We present a novel framework, dubbed PanoVerse, for the automatic creation and presentation of immersive stereoscopic environments from a single indoor panoramic image. Once per 360^CIRC shot, a novel data-driven architecture generates a fixed set of panoramic stereo pairs distributed around the current central view-point. Once per frame, directly on the HMD, we rapidly fuse the precomputed views to seamlessly cover the exploration workspace. To realize this system, we introduce several novel techniques that combine and extend state-of-the art data-driven techniques. In particular, we present a gated architecture for panoramic monocular depth estimation and, starting from the re-projection of visible pixels based on predicted depth, we exploit the same gated architecture for inpainting the occluded and disoccluded areas, introducing a mixed GAN with self-supervised loss to evaluate the stereoscopic consistency of the generated images. At interactive rates, we interpolate precomputed panoramas to produce photorealistic stereoscopic views in a lightweight WebXR viewer. The system works on a variety of available VR headsets and can serve as a base component for Metaverse applications. We demonstrate our technology on several indoor scenes from publicly available data.

@InProceedings{PJHGSA23,
  author       = {Pintore, G. and Jaspe Villanueva, A. and Hadwiger, M. and Gobbetti, E. and Schneider, J. and Agus, M.},
  title        = {PanoVerse: automatic generation of stereoscopic environments from single indoor panoramic images for Metaverse applications},
  booktitle    = {Proc. Web3D 2023 - 28th International ACM Conference on 3D Web Technology},
  month        = {october},
  year         = {2023},
  note         = {Honorable mention award in the best paper category at Web3D 2023},
  keywords     = {visual computing},
  doi          = {10.1145/3611314.361591},
  url          = {https://publications.crs4.it/pubdocs/2023/PJHGSA23},
}