Deep Reflectance Fields
High-Quality Facial Reflectance Field Inference From Color Gradient Illumination
*This work was conducted at Google
Abstract
Photo-realistic relighting of human faces is a highly sought after feature with many applications ranging from visual effects to truly immersive virtual experiences. Despite tremendous technological advances in the field, humans are often capable of distinguishing real faces from synthetic renders. Photo-realistically relighting any human face is indeed a challenge with many difficulties going from modelling sub-surface scattering and blood flow to estimating the interaction between light and individual strands of hair. We introduce the first system that combines the ability to deal with dynamic performances to the realism of 4D reflectance fields, enabling photo-realistic relighting of non-static faces. The core of our method consists of a Deep Neural network that is able to predict full 4D reflectance fields from two images captured under spherical gradient illumination. Extensive experiments not only show that two images under spherical gradient illumination can be easily captured in real time, but also that these particular images contain all the information needed to estimate the full reflectance field, including specularities and high frequency details. Finally, side by side comparisons demonstrate that the proposed system outperforms the current state-of-the-art in terms of realism and speed.
Downloads
[© Copyrights by the Authors, 2019. This is the author's version of the work. It is posted here for your personal use. Not for redistribution.]
Citation
@inproceedings{Meka:2019,
author = {Meka, Abhimitra and Haene, Christian and Pandey, Rohit and Zollhoefer, Michael and Fanello, Sean and Fyffe, Graham and Kowdle, Adarsh and Yu, Xueming and Busch, Jay and Dourgarian, Jason and Denny, Peter and Bouaziz, Sofien and Lincoln, Peter and Whalen, Matt and Harvey, Geoff and Taylor, Jonathan and Izadi, Shahram and Tagliasacchi, Andrea and Debevec, Paul and Theobalt, Christian and Valentin, Julien and Rhemann, Christoph},
title = {Deep Reflectance Fields - High-Quality Facial Reflectance Field Inference From Color Gradient Illumination},
journal = {ACM Transactions on Graphics (Proceedings SIGGRAPH)},
url = {http://gvv.mpi-inf.mpg.de/projects/DeepReflectanceFields/},
volume = {38},
number = {4},
month = {July},
year = {2019},
doi = {10.1145/3306346.3323027},
}
Acknowledgments
The authors would like to thank all participants of the lightstage recordings. We also thank the authors of Yamaguchi et. al. [2018] and Shu et. al. [2017] for providing the results of their methods on our data. Christian Theobalt was supported by the ERC Consolidator Grants 4DRepLy (770784). Michael Zollhoefer was supported by the Max Planck Center for Visual Computing and Communications.