Event
WorldOracle World Model Evaluation Platform Opens for Submissions
5 reports · 5 sources
A research team from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA, and AgiBot proposed WorldOracle, an action-driven world model evaluation benchmark, and opened evaluation submissions to developers on October 16. The platform uses identical initial observations and low-level action sequences to test whether model predictions can reproduce real-world physical consequences and task state transitions. It covers two robot embodiments, AgiBot G2 and YAM, as well as first-person human hand manipulation scenarios, and supports participation by both open-weight and closed-source models. The evaluation data contains 335 real manipulation trajectories, each about 2 minutes long, of which roughly 40% are failed or suboptimal executions such as missed grasps, drops, and spilled water. The team's experiments on seven representative models show that some models can reproduce robotic arm motions relatively accurately, but their predictions of object states have large deviations, and long-horizon prediction performance drops markedly. The dataset, evaluation code, and leaderboard have been made public. The team also notes that for fluids, deformable objects, and stochastic contacts, the same action may produce multiple plausible outcomes, and a single reference cannot fully describe these possibilities.
Developments
- WorldOracle releases action-conditioned world model benchmark; seven models show deviations between predictions and real execution · 4 reports
- WorldOracle: Benchmarking action-conditioned world models against real-world executions · 1 reports
Report timeline
- WorldOracle releases action-conditioned world model benchmark; seven models show deviations between predictions and real execution
Princeton University, UT Austin, the University of Toronto, NVIDIA, and AgiBot jointly launched WorldOracle, an action-conditioned world model benchmark that uses the same initial frame and action sequence to compare model predictions with real execution.
The Robot Report ·
- AgiBot, NVIDIA, and universities launch WorldOracle world-model evaluation platform
WorldOracle, a world-model evaluation platform jointly developed by researchers from AgiBot, NVIDIA, and several universities, is now live and open for evaluation submissions. The platform uses 335 real manipulation trajectories, with about 40% of interaction segments containing failed or suboptimal operations, testing model prediction against real execution outcomes. Testing of 7 world models found that while some models can reproduce robotic-arm motions relatively accurately, their predictions of object states still show considerable deviation, and performance drops markedly on long-horizon prediction.
36Kr Embodied Intelligence ·
- WorldOracle world-model evaluation platform launched, using real-robot testing to fact-check the industry
On October 16, WorldOracle, an action-driven world-model evaluation platform jointly completed by researchers from AgiBot, NVIDIA, Princeton University, the University of Texas at Austin, and the University of Toronto, was officially released, opening evaluation submissions to developers.
雷科技 ·
- WorldOracle: Benchmarking action-conditioned world models against real-world executions
WorldOracle is a benchmark for action-conditioned world models (ACWMs) that uses identical initial observations and low-level action sequences to test whether predicted rollouts can reproduce real-world physical consequences and task-state transitions. The evaluation covers four dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence, and is paired with long, multi-segment real-world data containing both successful and failed executions. Experiments on seven representative ACWMs reveal different models' strengths and failure modes. The dataset, evaluation code, and leaderboard are publicly available.
arXiv cs.RO ·
- World model evaluation platform WorldOracle opens submissions, first round tests seven models
WorldOracle officially opened submissions to developers on October 16. It is an evaluation platform that uses real execution records as a reference to test the prediction capabilities of action-driven world models. The paper is titled "WorldOracle."
机器之心 · 微信公众号 ·