Event
SpatialDex pretrains bimanual dexterous manipulation using human videos
1 reports · 1 sources
Xingyi He and other researchers propose the SpatialDex framework, which learns 3D-grounded bimanual dexterous manipulation from pure RGB human videos: for each video, it reconstructs the scene geometry and estimates hand motion, aligns the two at the video-level scale using a hand-size prior from 3D hand estimation, and then uses the reconstructed geometry both as policy input and as a prediction target. The method is pretrained on 1 million human video clips and then fine-tuned with a small number of robot demonstrations. The authors report that it scores 16.5 points higher than existing methods on nine simulated dexterous tasks across two embodiments, and 38.3 points higher on six real-world bimanual dexterous tasks; under matched comparisons, pretraining with reconstructed geometry achieves a 14.2-point higher success rate than pretraining with hand motion alone.
Report timeline
- SpatialDex: Pre-training 3D-Grounded Bimanual Dexterous Manipulation from RGB-Only Human Videos
SpatialDex is a framework that learns 3D-grounded bimanual dexterous manipulation from RGB-only human videos through reconstruction-augmented visuomotor pre-training, learning human behavior together with its spatial context. It reconstructs scene geometry and estimates hand motion from each video, aligns the two at a video-level scale using the hand-size prior of 3D hand estimates, and uses the reconstructed geometry as both a policy input and a prediction target.
arXiv cs.RO ·