Event
RoboJev-27B Open-Weight Vision-Language Judgment Model Released
1 reports · 1 sources
Josef Liyanjun Chen released the open-weight vision-language model RoboJev-27B along with inference code. The model takes a task instruction and 10 frames of operation footage (and, if an auxiliary camera is present, its final frame) as input, and on that basis predicts whether the robot has completed the task. On 268 held-out clips from the same model of SO-101, the model achieved 95.2% balanced accuracy and 0.993 AUROC; on 459 clips from unseen laboratories or robots, AUROC was 0.890. Median inference time for a single clip on a single H100 was 0.26 seconds. The model was trained with rank-32 LoRA adapters applied only to the language model portion, with the vision encoder kept frozen, and the training data consisted of 52,598 episode records. The release also publishes 20,628 predictions, organized under 29 model/protocol entries, for reproducing the benchmark tests. The author notes that comparisons with other methods are not strictly under identical conditions, because RoboJev-27B received more frames at inference time.
Report timeline
- RoboJev-27B: An Open-Source Vision-Language Model for Judging Robot Task Success
RoboJev-27B is an open-source vision-language model that takes a task instruction and 10 frames of a manipulation attempt as input and predicts whether the robot completed the task. It achieves 95.2% balanced accuracy and 0.993 AUROC on 268 held-out episodes of the same robot model (SO-101), and an AUROC of 0.890 on 459 episodes from unseen labs or robots. Median inference time is 0.26 seconds per episode on a single H100, and the model, inference code, and evaluation results for 20,628 predictions are released together.
arXiv cs.RO ·