Download Video: HD (MP4, 73 MB)


We propose the first algorithm to automatically and jointly synthesize both, the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architecture that leverages the inherent correlation between the facial expression and hand gestures. Synthesis of conversational body gestures is a multi-modal problem since many similar gestures can plausibly accompany the same input speech. To synthesize plausible body gestures in this setting, we train a Generative Adversarial Network (GAN) based model that measures the plausibility of the generated sequences of 3D body motion when paired with the input audio features. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people. To this end, we apply state-of-the-art monocular approaches for 3D body and hand pose estimation as well as dense 3D face performance capture to the video corpus. In this way, we can train an orders of magnitude more data than previous algorithms resorting to complex in-studio motion capture solutions, and thereby train much more expressive synthesis algorithms. Our experiments and user study show the state-of-the-art quality of our speech-synthesized full 3D character animations.



BibTeX, 1 KB

Author = {Habibie, Ikhsanul and Xu, Weipeng and Mehta, Dushyant and Liu, Lingjie and Seidel, Hans-Peter and Pons-Moll, Gerard and Elgharib, Mohamed and Theobalt, Christian},
Title = {Learning Speech-driven 3D Conversational Gestures from Video},
Booktitle = {ACM International Conference on Intelligent Virtual Agents (IVA)},
Year = {2021},
Eprint = {Todo},


This work was supported by the ERC Consolidator Grant 4DRepLy (770784).
Gerard Pons-Moll is funded by the Deutsche Forschungsgemeinschaft (DFG. German Research Foundation) - 409792180.


For questions, clarifications, please get in touch with:
Ikhsanul Habibie

This page is Zotero translator friendly. Page last updated Imprint. Data Protection.