Event
Labference: A Robotic Laboratory Reasoning-Action Benchmark Released
1 reports · 1 sources
Shitong Liu and other authors propose the Labference benchmark, which evaluates how robots adjust subsequent operations based on actual physical execution results, and separately measures manipulation selection, physical execution, and task completion. The benchmark tests outcome-dependent behavioral adaptation through controlled evidence changes, and uses Oracle-assisted continuation with shared checkpoints to evaluate which kinds of support can improve subsequent choices and task completion. Initial evaluations show that the tested VLA policies still make target-selection errors even when object transfer succeeds; GPT-driven control, although able to select correctly, encounters execution failures and higher computational overhead.
Report timeline
- Labference: A Benchmark for Evaluating the Reasoning-Action Connection in Robotic Laboratories
The authors propose Labference, a benchmark for evaluating how robots adjust subsequent operations based on actual physical execution outcomes, separately measuring operation selection, physical execution, and task completion. Preliminary evaluation shows that the tested VLA policies still made target-selection errors even when object transfer succeeded, while GPT-driven control, though able to select correctly, encountered execution failures and higher computational cost.
arXiv cs.RO ·