ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting

ICCV 2023 (Oral)

Ruofan Liang1,2, Huiting Chen1, Chunlin Li1, Fan Chen1 Selvakumar Panneer3 Nandita Vijaykumar1,2

1 University of Toronto   2 Vector Institute   3 Intel Labs  

Paper

Code

Colab

Abstract

Envidr

Recent advances in neural rendering have shown great potential for reconstructing scenes from multiview images.However, accurately representing objects with glossy surfaces remains a challenge for existing methods. In this work, we introduce ENVIDR, a rendering and modeling framework for high-quality rendering and reconstruction of surfaces with challenging specular reflections. To achieve this, we first propose a novel neural renderer with decomposed rendering components to learn the interaction between surface and environment lighting. This renderer is trained using existing physically based renderers and is decoupled from actual scene representations. We then propose an SDF-based neural surface model that leverages this learned neural renderer to represent general scenes. Our model additionally synthesizes indirect illuminations caused by inter-reflections from shiny surfaces by marching surface-reflected rays. We demonstrate that our method outperforms state-of-art methods on challenging shiny scenes, providing high-quality rendering of specular reflections while also enabling material editing and scene relighting.


Overview

Pipeline
Our neural renderer learns an approximation of physically based rendering (PBR) using 3 decomposed MLPs accounting for environment lighting, diffuse rendering, and specular rendering, respectively. Each Env. MLP represents one speficic environment light probe. We utilize Filament PBR engine to synthesize training images during runtime. The pre-trained diffuse and specular MLPs will be fixed and integrated into implicit surface models for reconstructing and rendering general scenes. A simple demo for rendering spheres is available in a Colab Notebook.

Results


Misc

  • Check out a nice document from Filament to learn more about the basics of physically-based rendering (PBR).

Citation


@article{liang2023envidr,
  title={ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting},
  author={Liang, Ruofan and Chen, Huiting and Li, Chunlin and Chen, Fan and Panneer, Selvakumar and Vijaykumar, Nandita},
  journal={arXiv preprint arXiv:2303.13022},
  year={2023}
}