Event
Evaluation: Spatial instruction rewording reduces stable placement success rate by 12.2 percentage points
1 reports · 1 sources
Ali-Adeeb Abbas et al. conducted 864 rollouts of three world-action models across four RoboLab scenarios, fixing initial states and task goals, to evaluate consistency under equivalent spatial instruction rewording. The results show that describing spatial relations using a reference object rather than the object to be moved reduces the proportion achieving stable placement by an average of 12.2 percentage points. An independent test of policy language understanding identified relational reversals and showed that adding images interferes with otherwise correct text-only interpretations. Rollout analysis further distinguished spatial placement from instruction following: moving the wrong object may still satisfy the required spatial relation, while the stable placement rate drops from 43.2% at any timestep to 21.8% at the end of the episode.
Report timeline
- Evaluating Sensitivity of World-Action Models to Spatial Instruction Rewriting
An evaluation conducted 864 rollouts of three world-action models across four RoboLab scenes, holding initial states and task goals fixed, and found that describing spatial relations using the reference object rather than the object to be moved lowered the stable placement success rate by an average of 12.2 percentage points. An independent test of the policies' language understanding identified relation reversals, and adding images disrupted otherwise correct text-only interpretations. Rollout analysis also distinguished spatial placement from instruction following: the stable placement rate fell from 43.2% at any time to 21.8% at episode end.
arXiv cs.RO ·