Graphics, Vision & Video

Real-time Non-rigid Reconstruction using an RGB-D Camera

ACM Transactions on Graphics 2014 (TOG)

Michael Zollhöfer 1   Matthias Nießner 2   Shahram Izadi 3   Christoph Rhemann 3   Christopher Zach 3
Matthew Fisher 2   Chenglei Wu 4   Andrew Fitzgibbon 3   Charles Loop 3   Christian Theobalt 4   Marc Stamminger 1
1 University of Erlangen-Nuremberg 2 Stanford University 3 Microsoft Research 4 MPI for Informatics
Abstract Videos Bibtex

Abstract

We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time. Our system uses a single self-contained stereo camera unit built from off-the-shelf components and consumer graphics hardware to generate spatio-temporally coherent 3D models at 30 Hz. A new stereo matching algorithm estimates real-time RGB-D data. We start by scanning a smooth template model of the subject as they move rigidly. This geometric surface prior avoids strong scene assumptions, such as a kinematic human skeleton or a parametric shape model. Next, a novel GPU pipeline performs non-rigid registration of live RGB-D data to the smooth template using an extended non-linear as-rigid-as-possible (ARAP) framework. High-frequency details are fused onto the final mesh using a linear deformation model. The system is an order of magnitude faster than state-of-the-art methods, while matching the quality and robustness of many offline algorithms. We show precise real-time reconstructions of diverse scenes, including: large deformations of users' heads, hands, and upper bodies; fine-scale wrinkles and folds of skin and clothing; and non-rigid interactions performed by users on flexible objects such as toys. We demonstrate how acquired models can be used for many interactive scenarios, including re-texturing, online performance capture and preview, and real-time shape and motion re-targeting

Paper
pdf (3.3M) 		Video
avi (206M)		 Presentation
not yet available

Videos

Supplementary video to the paper

Bibtex

  
@article{zollhoefer2014deformable,	
title={Real-time Non-rigid Reconstruction using an RGB-D Camera},	
author={Zollh{\"o}fer, Michael and Nie{\ss}ner, Matthias and Izadi, Shahram and Rhemann, Christoph and Zach, Christopher and Fisher, 
Matthew and Wu, Chenglei and Fitzgibbon, Andrew and Loop, Charles and Theobalt, Christian and Stamminger, Marc}, 
journal = {ACM Transactions on Graphics (TOG)},	
publisher = {ACM},	
volume = {33},	
number = {4},	
year = {2014} 
}