Deep Relightable Textures

Volumetric Performance Capture with Neural Rendering



1Google   2Max Planck Institute for Informatics   3Saarland Informatics Campus   
*Authors contributed equally to this work

This work was conducted at Google

Abstract


The increasing demand for 3D content in augmented and virtual reality has motivated the development of volumetric performance capture systems such as the Light Stage. Recent advances are pushing free viewpoint relightable videos of dynamic human performances closer to photorealistic quality. However, despite significant efforts, these sophisticated systems are limited by reconstruction and rendering algorithms which do not fully model complex 3D structures and higher order light transport effects such as global illumination and sub-surface scattering. In this paper, we propose a system that combines traditional geometric pipelines with a neural rendering scheme to generate photorealistic renderings of dynamic performances under desired viewpoint and lighting. Our system leverages deep neural networks that model the classical rendering process to learn implicit features that represent the view-dependent appearance of the subject independent of the geometry layout, allowing for generalization to unseen subject poses and even novel subject identity. Detailed experiments and comparisons demonstrate the efficacy and versatility of our method to generate high-quality results, significantly outperforming the existing state-of-the-art solutions.

Downloads




[© Copyrights by the Authors, 2020. This is the author's version of the work. It is posted here for your personal use. Not for redistribution.]

Citation

BibTeX, 1 KB

@inproceedings{Meka:2020,
	author = {Meka, Abhimitra and Pandey, Rohit and Haene, Christian and Orts-Escolano, Sergio and Barnum, Peter and Davidson, Philip and Erickson, Daniel and Zhang, Yinda and Taylor, Jonathan and Bouaziz, Sofien and Legendre, Chloe and Ma, Wan-Chun and Overbeck, Ryan and Beeler, Thabo and Debevec, Paul and Izadi, Shahram and Theobalt, Christian and Rhemann, Christoph and Fanello, Sean},
	title = {Deep Relightable Textures - Volumetric Performance Capture with Neural Rendering},
	journal = {ACM Transactions on Graphics (Proceedings SIGGRAPH Asia)},
	url = {http://gvv.mpi-inf.mpg.de/projects/DeepRelightableTextures/},
	volume = {39},
	number = {6},
	month = {December},
	year = {2020},
	doi = {10.1145/3414685.3417814},
}
	

Acknowledgments

The authors would like to thank all participants of the Light Stage recordings. We also thank the authors of Thies et. al. [2019] for providing the implementation of their method for comparisons. Christian Theobalt was supported by the ERC Consolidator Grants 4DRepLy (770784).

Contact

Abhimitra Meka
ameka@mpi-inf.mpg.de