We introduce the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks. Compared to the competing methods, our combination of loss functions is fully-differentiable and can be readily integrated into deep-learning systems. We formulate the deformation model by an auto-decoder and impose subspace constraints on the recovered latent space function in a frequency domain. Thanks to the state recurrence cue, we classify the reconstructed non-rigid surfaces based on their similarity and recover the period of the input sequence. Our N-NRSfM approach achieves competitive accuracy on widely-used benchmark sequences and high visual quality on various real videos. Apart from being a standalone technique, our method enables multiple applications including shape compression, completion and interpolation, among others. Combined with an encoder trained directly on 2D images, we perform scenario-specific monocular 3D shape reconstruction at interactive frame rates. To facilitate the reproducibility of the results and boost the new research direction, we open-source our code and provide trained models for research purposes.


ECCV 2020 Videos

  • Highlights (1 min, 41 Mb)

  • Talk (10 mins, 142 Mb)


BibTeX, 1 KB

       author = {Sidhu, Vikramjit and Tretschk, Edgar and Golyanik, Vladislav and Agudo, Antonio and Theobalt, Christian}, 
       title = {Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints}, 
       booktitle = {European Conference on Computer Vision (ECCV)}, 
       year = {2020} 


Supported by the ERC Consolidator Grant 4DReply (770784) and the Spanish Ministry of Science and Innovation under project HuMoUR TIN2017-90086-R.


For questions, clarifications, please get in touch with:
Edgar Tretschk tretschk@mpi-inf.mpg.de
Vladislav Golyanik golyanik@mpi-inf.mpg.de

This page is Zotero translator friendly. Page last updated Imprint. Data Protection.