Event

BEHAVIOR-Spatial robot spatial understanding benchmark released

1 reports · 1 sources

The research team proposed and released the BEHAVIOR-Spatial benchmark, built on BEHAVIOR-1K, to evaluate the relationship between the spatial understanding of vision-language models (VLMs) and vision-language-action (VLA) behavior. The benchmark contains 3,256 spatial questions across 49 activities, 1,240 grounding queries across 14 manipulation tasks, and action evaluations covering 14 manipulation tasks and 10 navigation tasks. In the manipulation task evaluations, the baseline achieved a success rate of 41.6% on held-out configurations, but when the required target changed within a fixed training layout, the success rate dropped to 16.1%. The authors state that gains from VLM adaptation and robot data pretraining were limited, while providing oracle visual target cues raised the two success rates to 60.3% and 38.0%, respectively.