RoboTryOn: Synthesizing Robot Dexterous Manipulation Demonstrations from Human Videos via Virtual Try-On

RoboTryOn proposes treating human-to-robot translation as virtual try-on: it preserves the demonstrated motion, objects, and background, and only replaces the human with the target robot specified by a reference image. The method autoregressively converts keyframes and uses them to guide video synthesis, with motion conditioned on the human hand skeleton; it then uses an inverse dynamics model (IDM) with depth features to infer action labels from the synthesized videos for policy learning. Experiments show that policies trained only on synthetic data improve average success rate from 2% to 60% on new tasks, including dual-arm manipulation, and reach the level of policies taught by pure teleoperation at about one-third of the collection cost.