[Monocular Reconstruction of Neural Face Reflectance Fields]

Monocular Reconstruction of Neural Face Reflectance Fields

Mallikarjun B R1,  Ayush Tewari1,  Tae-Hyun Oh2,  Tim Weyrich3,  Bernd Bickel4,  Hans-Peter Seidel1,  Hanspeter Pfister5,  Wojciech Matusik6,  Mohamed Elgharib1,  Christian Theobalt1

1 Max Planck Institute for Informatics, Saarbrücken, Germany
2 POSTECH
3 University College London
4 Institute for Science and Technology Austria
5 Harvard University
6 Massachusetts Institute of Technology

Abstract

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance as higher-order global illumination effects and self-shadowing are not modeled. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage training dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods, in terms of photorealism due to better capturing of physical premitives, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.

Citation Style:    Publication

Monocular Reconstruction of Neural Face Reflectance Fields.
Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt.
Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.

Related Publications and Reports

[PhotoApp: Photorealistic Appearance Editing of Head Portraits]
PhotoApp: Photorealistic Appearance Editing of Head Portraits.
Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, Christian Theobalt.
Transactions on Computer Graphics (Proc. SIGGRAPH), to appear, August 2021.
Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, and Christian Theobalt. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH), August 2021.B R, M., Tewari, A., Dib, A., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Chevallier, L., Elgharib, M., and Theobalt, C. 2021. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH).M. B R, A. Tewari, A. Dib, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, L. Chevallier, M. Elgharib, and C. Theobalt, “PhotoApp: Photorealistic appearance editing of head portraits,” in Transactions on Graphics (Proc. SIGGRAPH), Aug. 2021.
[Web Page][Preprint PDF (14 MB)][BibTeX]
[Analysis of Human Faces using a Measurement-Based Skin Reflectance Model]
Analysis of Human Faces using a Measurement-Based Skin Reflectance Model.
Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, Markus Gross.
ACM Transactions on Graphics (Proc. SIGGRAPH), pp. 1013–1024, Boston, MA, July 2006.
Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006), 25(3):1013–1024, 2006.Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006) 25, 3, 1013–1024.T. Weyrich, W. Matusik, H. Pfister, B. Bickel, C. Donner, C. Tu, J. McAndless, J. Lee, A. Ngan, H. W. Jensen, and M. Gross, “Analysis of human faces using a measurement-based skin reflectance model,” ACM Trans. on Graphics (Proc. SIGGRAPH 2006), vol. 25, no. 3, pp. 1013–1024, 2006.
[Web Page][PDF][Video][MERL/ETH Skin Reflectance Database][BibTeX]

Acknowledgments

This work was supported by the ERC Consolidator Grant 4DReply (770784).


Disclaimer: The documents contained in these pages are included to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.