Event
A model for co-generating 3D motion and video for arbitrary skeletons
2 reports · 2 sources
Researchers including Noam Rotstein propose the "Joints, Camera, Action" model, which can simultaneously generate video and replayable 3D motion for any given skeleton, and output a camera that registers the motion to the scene. Paper evaluations show that the model generates diverse skeleton animations from text more faithfully than video-first pipelines, and that the motion responds to the surrounding scene. The model keeps 3D motion in a separate module to support arbitrary skeletons, connecting it to a pretrained video model through a small number of cross-attention layers; the motion model first learns skeleton motion from pure motion data, then combines human and robot videos and uses a generative video editor to curate nearly 200,000 training pairs. The model also supports video-to-skeleton motion, skeleton motion-to-video, and keyframe-based 3D motion control.
Report timeline
- Joints, Camera, Action! A Co-Generation Model for 3D Motion and Video for Arbitrary Skeletons
The paper proposes a model that can simultaneously co-generate video and 3D motion for any given skeleton, along with a camera that registers that motion to the scene. The model keeps 3D motion in a separate module to support arbitrary skeletons, connects it to a pretrained video model through a small number of cross-attention layers, and uses a generative video editor to curate nearly 200K training pairs. Evaluations show that its text-driven animation of diverse skeletons is more faithful than video-first pipelines, that the motion responds to the surrounding scene, and that it also supports video-to-skeleton motion, skeleton motion-to-video, and keyframe-based 3D motion control.
arXiv cs.RO ·
- JCA model jointly generates 3D motion and video for arbitrary skeletons
An NVIDIA research team proposes the JCA model, which can simultaneously generate video and replayable 3D motion for any given skeleton, and output a camera that registers the motion to the scene. The model keeps 3D motion in a separate model that supports arbitrary skeletons, connects to a pretrained video model through a small number of cross-attention layers, and the motion model first learns skeleton movement from motion-only data.
HuggingFace Daily Papers ·