Event
Execution logs can sway VLM judgments of robot progress
1 reports · 1 sources
Research by Weiyan Zhao and Sunny Yang shows that when vision-language models evaluate robot videos, text placed alongside the footage can change their conclusions. After adding a single log line saying "making progress" to clips in which the robot was lagging behind, the share of correct conclusions that Gemini, Qwen and GPT judges flipped to "progress" was 10 to 35 percentage points higher than when neutral text was added. The study also found that adding a note saying the logs may be wrong eliminated most of this effect; and partially fabricated "task completed" claims caused Qwen to flip half of correct conclusions, while irrelevant log lines caused only about one-tenth of the flips.
Report timeline
- Execution logs sway VLM judgments of robot progress
The study tested how text placed beside frames affects vision-language models judging robot videos: when a line saying "making progress" was added to clips where the robot was falling behind, Gemini, Qwen, and GPT judges flipped correct verdicts to "progress" 10 to 35 percentage points more often than with neutral text.
arXiv cs.RO ·