Event

Release of RoboReel: A Benchmark for Evaluating Manipulation Skill Learning from Human Videos

1 reports · 1 sources

Weiwei Gu and other researchers propose RoboReel, a unified benchmark for evaluating Learning-from-Observation (LfO) models that learn policies from human videos. It contains real human demonstration videos, simulated robot trajectories, and evaluation environments for 10 manipulation tasks, and uses four test suites to evaluate models along dimensions such as robustness to visual distractors and long-horizon task completion ability. The researchers used this benchmark to evaluate more than 7 LfO algorithms (including their VLA-based variants), and reproduced some tasks in the real world, stating that the relative performance trends observed in simulation were maintained in the real world. The analysis shows that long-horizon tasks and low-tolerance tasks remain challenging for current models.