3D Motion Reconstruction from 2D Motion Data Using Multimodal Conditional Deep Belief Network

In this paper, we propose a deep generative model named Multimodal Conditional Deep Belief Network (MCDBN) for cross-modal learning of 3D motion data and their non-injective 2D projections on the image plane. This model has a three sectional structure, which learns conditional probability distribution of 3D motion data given 2D projections. Two distinct Conditional Deep Belief Networks (CDBNs), encode the real-valued spatiotemporal patterns of 2D and 3D motion time series captured from subjects’ movements into the compact representations. The third part includes a Multimodal Restricted Boltzmann Machines (MRBMs) which in the training process, learns the relationship between the compact representations of data modalities by variation information criteria. As a result, conditioned on a 2D motion data obtained from a video, MCDBN can regenerate 3D motion data in generation phase. We introduce Pearson correlation coefficient of ground truth and regenerated motion signals as a new evaluation metric in motion reconstruction problems. The model is trained with human motion capture data and the results show that the real and the regenerated signals are highly correlated which means the model can reproduce dynamical patterns of motion accurately.