Event
Proposes VLA memory intervention evaluation protocol IMEP and ChainBench
1 reports · 1 sources
Keyuan Luo et al. propose on arXiv an intervention-based memory evaluation protocol IMEP and benchmark ChainBench for vision-language-action (VLA) systems, used to determine whether memory truly affects robot decision-making, covering four questions: net benefit of memory, conditions under which it takes effect, whether it changes task outcomes, and whether the effect is specific to task-relevant facts, and evaluate within-episode, cross-episode, and agent memory. In cross-episode learned-bank experiments, intervening on "one unit at the last position" (ULP) (including the installation procedure) changed 11.9% of same-seed paired outcomes, whereas replacing key facts with retained facts showed no significant additional effect. ChainBench requires correct decisions to depend on historical information absent from the current observation; under shared recorded execution, the full memory configuration raised task success rate from 39.1% to 74.3%, on par with the oracle decision condition.
Report timeline
- Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
The paper proposes an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems, used to determine whether memory truly influences robot decision-making, covering four questions: the net benefit of memory, the conditions under which it takes effect, changes to task outcomes, and whether the effects are specific to task-relevant facts.
arXiv cs.RO ·