Article

Evaluating Sensitivity of World-Action Models to Spatial Instruction Rewriting

An evaluation conducted 864 rollouts of three world-action models across four RoboLab scenes, holding initial states and task goals fixed, and found that describing spatial relations using the reference object rather than the object to be moved lowered the stable placement success rate by an average of 12.2 percentage points. An independent test of the policies' language understanding identified relation reversals, and adding images disrupted otherwise correct text-only interpretations. Rollout analysis also distinguished spatial placement from instruction following: the stable placement rate fell from 43.2% at any time to 21.8% at episode end.

Part of the event: Evaluation: Spatial instruction rewording reduces stable placement success rate by 12.2 percentage points

This page provides a summary and source link only.