Article

Joints, Camera, Action! A Co-Generation Model for 3D Motion and Video for Arbitrary Skeletons

The paper proposes a model that can simultaneously co-generate video and 3D motion for any given skeleton, along with a camera that registers that motion to the scene. The model keeps 3D motion in a separate module to support arbitrary skeletons, connects it to a pretrained video model through a small number of cross-attention layers, and uses a generative video editor to curate nearly 200K training pairs. Evaluations show that its text-driven animation of diverse skeletons is more faithful than video-first pipelines, that the motion responds to the surrounding scene, and that it also supports video-to-skeleton motion, skeleton motion-to-video, and keyframe-based 3D motion control.

Part of the event: A model for co-generating 3D motion and video for arbitrary skeletons

This page provides a summary and source link only.