Event

Aether AI and UC San Diego propose robot evaluation benchmark RAB

1 reports · 1 sources

A research team from Aether AI and UC San Diego proposed RobotAgentBench (RAB), a comprehensive evaluation benchmark for open-world robot autonomy. Starting from real homes, it constructs 50 home layouts, 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 categories of open-ended scenarios, for systematically comparing standalone VLA/WAM policies with multiple robot agent architectures. The research team developed an agent-assisted, human-in-the-loop-reviewed real-to-sim pipeline: starting from real home physical spaces, reconstructing scenes in simulation, then extending them into different interactive environment configurations, and collecting demonstration data required for training and evaluation. The team also evaluated InternVLA-A1.5, OpenWAM-Alpha, and the Codex full-code configuration using a wheeled humanoid robot on a matched subset of real homes and corresponding simulated environments, while matching the number of training demonstrations. The research team states that improved open-scenario performance remains limited, and how self-improvement translates into broader, more reliable autonomy still requires further research.