Event
Release of RoboJudge embodied video generation evaluation benchmark and model
1 reports · 1 sources
The author team led by Siyuan Ma proposes RoboJudge-Bench to evaluate the reliability of multimodal large language models as judges for embodied video generation, and fine-tunes a 9B evaluation model, RoboJudge-9B, on this dataset. The paper states that RoboJudge-9B achieves a Pearson correlation coefficient of 0.719 with human annotations, 0.089 higher than the best-performing closed-source baseline, Gemini-3.7-Flash. The benchmark covers two dimensions, physical consistency and instruction alignment, and contains 800 human-annotated videos from 8 source corpora and 10 video generators, covering grippers, human hands, and dexterous hands.
Report timeline
- RoboJudge-Bench evaluates multimodal large models as judges of embodied video generation
The paper introduces RoboJudge-Bench to assess the reliability of multimodal large language models (MLLMs) as judges of embodied video generation, covering two dimensions, physical consistency and instruction alignment, with 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands.
arXiv cs.RO ·