Event

COWBOY framework achieves zero-shot transfer for whole-body dexterous grasping

1 reports · 1 sources

Jiahui Yang et al. propose the COWBOY framework, which distills a whole-body controller and local reinforcement learning experts into a single vision policy, achieving zero-shot transfer to hundreds of objects and various in-the-wild scenarios on whole-body dexterous grasping tasks, including cluttered tabletops, shelves, drawers, and other unstructured environments. The framework does not directly use reinforcement learning to learn scene adaptation in simulation; instead, it uses a vectorized environment-aware whole-body controller to handle reaching motions across thousands of scenes, coordinating base-arm movement, collision avoidance, and active vision; local reinforcement learning experts are trained on object interaction in local scenes and can be reused across thousands of full scene layouts. The final distilled single vision policy controls the mobile base, robotic arm, dexterous hand, and actuated camera from point cloud observations.