Abstract
Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational l2– lp optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality – even without user guidance such as scribbles.
Downloads
[© Copyrights by the Authors, 2016. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version will be published in ACM Transactions on Graphics, http://dx.doi.org/10.1145/2897824.2925907]
Citation
@article{meka:2016, author = {Meka, Abhimitra and Zollh{\"o}fer, Michael and Richardt, Christian and Theobalt, Christian} title = {Live Intrinsic Video}, journal = {ACM Transactions on Graphics (Proceedings SIGGRAPH)}, url = {http://gvv.mpi-inf.mpg.de/projects/LiveIntrinsicVideo/}, volume = {35}, number = {4}, year = {2016}, doi = {10.1145/2897824.2925907}, keywords = {intrinsic decomposition, reflectance, shading, p-norm, real time, data-parallel optimization, recoloring, retexturing}}
Acknowledgments
This research was funded by the ERC Starting Grant CapReal (335545). We thank Nicolas Bonneel and Yebin Liu for providing the data for comparisons to their state-of-the-art techniques.