We present BimArt, a novel generative approach for synthesizing 3D bimanual hand interactions with articulated objects. Unlike prior works, we do not rely on a reference grasp, a coarse hand trajectory, or separate modes for grasping and articulating. To achieve this, we first generate distance-based contact maps conditioned on the object trajectory with an articulation-aware feature representation, revealing rich bimanual patterns for manipulation. The learned contact prior is then used to guide our hand motion generator, producing diverse and realistic bimanual motions for object movement and articulation. Our work offers key insights into feature representation and contact prior for articulated objects, demonstrating their effectiveness in taming the complex, high-dimensional space of bimanual hand-object interactions. Through comprehensive quantitative experiments, we demonstrate a clear step towards simplified and high-quality hand-object animations that excel over the state-of-the-art in motion quality and diversity.
BimArt takes N frames of object trajectories as input and generates N frames of 3D bimanual interactions. The object features (articulation-aware BPS features O, 6D global states G, and the object scale so) are passed into both the object encoder Eo (MLP) in the contact generation model and Eα (MLP) in the motion generation model. Additionally, the motion generation model’s contact encoder Ec takes C, the bimanual contact map produced by the contact generation model, as conditioning input. The contact model and motion model are both denoising diffusion models, and the spiral denotes the denoising process. C is further used as guidance at each diffusion timestep to align hand motions with the generated contact maps. Finally, we use optimization to correct contact and penetration artifacts and obtain 3D bimanual meshes.
@article{zhang2024bimart,
title = {BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects},
author = {Zhang, Wanyue and Dabral, Rishabh and Golyanik, Vladislav and Choutas, Vasileios and
Alvarado, Eduardo and Beeler, Thabo and Habermann, Marc and Theobalt, Christian},
year = {2024},
journal={arXiv preprint arXiv:2412.05066}
}