Event
Generating thousands of robot demonstration data from a single human RGB video
1 reports · 1 sources
Matthew Strong et al. propose the Real2Sem2Real (R2Sem2R) pipeline, which converts a single monocular RGB human demonstration video into thousands of robot demonstration data for multiple semantic object instances within the same category, generating over 2,000 entries in about two hours. The method uses Gaussian splatting distilled from DINOv3 to track object SE(3) poses, outperforming existing methods and FoundationPose using estimated metric depth, and maps human hand motions into feasible robot grasps transferable to digital twins. Robot demonstrations are generated in IsaacSim (photorealistic rendering) or MuJoCo (contact physics). The authors claim that on two real robots and four tasks, training vision-language-action policies with one human video per task achieves performance on diverse object instances that meets or exceeds policies trained with teleoperation data.
Report timeline
- Real2Sem2Real: Generating Cross-Instance Robot Training Data from a Single Human RGB Demonstration
Real2Sem2Real (R2Sem2R) proposes a real-semantic-real pipeline that turns a single monocular RGB human demonstration video into thousands of robot demonstrations covering multiple semantic object instances within the same category. The method tracks object SE(3) poses using DINOv3-distilled Gaussian splats, outperforming existing methods and FoundationPose with estimated metric depth, and maps the human hand to a feasible robot grasp that can be transferred to digital twins.
arXiv cs.RO ·