Event
ViT-H: Direct Video-to-Humanoid Motion Generation
1 reports · 1 sources
Zimo He and co-authors released ViT-H, a video-to-humanoid control framework that adopts a "generate-then-track" approach: it uses a diffusion model to generate humanoid reference motions directly from monocular video, skipping the human motion reconstruction and retargeting stages, and post-trains the generator with direct preference optimization using motion tracking results. The authors report that experiments on the Unitree G1 show ViT-H outperforms existing reconstruction-retargeting-tracking pipelines in motion quality and executability, and that performance improves with model scale and with the scale of pretraining and post-training data. The authors also present an autoregressive variant, which they say can achieve real-time humanoid video teleoperation at under 400 ms latency.
Report timeline
- ViT-H: Generating humanoid motions directly from video with diffusion models and post-training via tracking feedback
ViT-H proposes a generation-plus-tracking video-to-humanoid control framework. It uses a diffusion model to generate humanoid reference motions directly from monocular video, skipping human motion reconstruction and retargeting, and uses motion-tracking results to post-train the generator with direct preference optimization. The authors also present an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Experiments on the Unitree G1 show that ViT-H outperforms existing reconstruction-retargeting-tracking pipelines in motion quality and executability, and that performance improves with model size and with pre-training and post-training data scale.
arXiv cs.RO ·