Content types
Benchmarks and evaluations
Test results, comparisons, reproductions, leaderboard changes, and disputes about evaluation methods for existing models, robots, or systems. New methods or benchmark designs belong under Research papers; promotional demonstrations are not automatically evaluations.
42 selected · 26 in the last 30 days
arXiv cs.RO ·
Paper analyzes agentic manipulation policies: AMP1 achieves 85.6% zero-shot success rate
The paper proposes agentic manipulation policies (AMP), in which a generalist AI model directly controls a robotic arm and adjusts its actions as the task progresses. The authors' reference implementation, AMP1, runs unchanged across single-arm, bimanual, and mobile manipulation.
Xbotics具身智能实验室 · 微信公众号 ·
GE-Sim 2.0 and π0.5 closed-loop simulation test: Why does reward hacking occur in VLA training?
The Xbotics Embodied Intelligence Lab built a closed-loop simulation pipeline using the AgiBot GE-Sim 2.0 world model and the OpenPI π0.5 policy. In testing, the world model generated about 4.33 frames per second, the pure-policy closed loop ran at about 3.20 frames per second, and after adding MiMo online scoring, the rate dropped to about 0.88 frames per second.
arXiv cs.RO ·
SimuGuard: Auditing Contact Artifacts in Simulation Benchmarks Such as LIBERO and RoboTwin
The authors propose SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa, and RoboDojo that identifies failures caused by the simulator rather than the policy. On RoboTwin, the authors trace these failures to the physics engine's collision detection: when geometry is 1 to 2 centimeters apart, the engine reports a penetration of about 2 centimeters, and the solver ejects the object at several meters per second because the distance algorithm discards triangles below a fixed area threshold.
arXiv cs.RO ·
From Teleoperation to Long-Horizon Dexterity: Evaluating Visuomotor Skills and Offline Proxies for Dexterous Manipulation
The research team used a five-fingered robotic hand to grasp and insert RJ45 connectors, evaluating 281 imitation-learning policies spanning seven policy families in approximately 1,700 real-robot trials. Controlled ablations showed that observation and action representations had a greater impact than conventional optimizer and sampling choices. For point-cloud policies, region-of-interest cropping, representation in the robot-base frame, and normalization within the point encoder each improved success by 70 to 100 percentage points. For the vision-language-action policy, switching to absolute action prediction improved insertion success by up to 70 percentage points.
机器之心 · 微信公众号 ·
Qiyuan releases PrimeSim world-model simulator, tops EWMBench and WorldSimProbe
The Qiyuan Embodied Research Center has released PrimeSim, a world-model simulator for robot manipulation tasks, built by adapting a pretrained video generation model through mid-training and post-training stages. It ranks first on EWMBench with an overall score of 6.09, and took first place in three of five WorldSimProbe evaluations: global trajectory coverage, action-source behavior preservation, and interaction dynamics.
机器之心 · 微信公众号 ·
AgiBot releases embodied model GO-3 Preview, enabling full-pipeline scaled learning from pretraining to post-deployment
AgiBot has released the embodied model GO-3 Preview, focused on full-pipeline scaled learning across pretraining, post-training, and post-deployment, and it ranked first in three evaluations: RoboDojo, RoboTwin, and RoboColiseum.
机器之心 · 微信公众号 ·
X-Era AI opensources 4D world action model E0.5, real-robot success rate surpasses π0.5
X-Era AI has open-sourced its native 4D world action model E0.5, achieving overall average success rates of 85.0% and 88.3% on the AgileX Piper dual-arm robot and the Zerith H1 humanoid robot, respectively, 13.3 and 26.7 percentage points higher than π0.5.
钛媒体 ·
Astra's Shockwave and the Embodied AI Industry's 50 Days
In the 50 days since GPT-6 Astra was released, the embodied AI industry has conducted intensive evaluations and discussions around it, and general-purpose large models have begun directly taking over robot operation. Tests by SenseTime and several institutions concluded that Astra is basically capable of completing embodied tasks, but still has a clear gap from practical use in real environments, underperforming in millimeter-level contact, dynamic control, and response speed. The article also reviews industry uses such as using Astra directly for policies, reviewing VLA actions, building simulation environments, and data annotation, as well as chain reactions including cooling financing and major companies stepping up data collection.
AgiBot ·
AgiBot releases embodied foundation model GO-3 Preview, ranks first on three embodied intelligence benchmarks
AgiBot has released the embodied foundation model GO-3 Preview, built on its self-developed WRAM (World Reasoning Action Model) architecture that unifies task understanding, future state prediction, and action generation, and it ranks first on three embodied intelligence benchmarks: RoboDojo, RoboTwin, and RoboColiseum. It is planned to be fully open-sourced by the end of 2026.
简智机器人 · X ·
GenRobot.AI's two open-source datasets enter Hugging Face robotics dataset trending top five
GenRobot.AI announced that both of its open-source datasets, Gen-HumanWholeBody and Gen-HumanEgo, have entered the top five of the Hugging Face robotics dataset trending list. Gen-HumanWholeBody is a first-person whole-body manipulation dataset containing 600 hours of whole-body manipulation data, 16,976 real-world scenes, and 1,281 human skills, with whole-body MPJPE of about 3 cm; the authors say it targets whole-body motion learning, human-to-robot transfer, and embodied foundation model research.
机器之心 · 微信公众号 ·
Tsinghua team builds HandsOn dexterous manipulation benchmark with 66 tasks to evaluate robot bimanual capabilities
Ding Ning's team at Tsinghua University, together with researchers from Beijing University of Posts and Telecommunications, Peking University, and Nature Will, built HandsOn, a simulation platform and evaluation benchmark for cross-embodiment dexterous manipulation, containing 66 manually verified tasks and 18,000 human simulation teleoperation demonstrations.
Xbotics具身智能实验室 · 微信公众号 ·
How to Choose a Robot Simulator: Same Success, Tenfold Contact Force Gap
The article interprets a simulator comparison report released by Manda Robotics in early October, running about ten sets of experiments with the same Franka 7-axis robotic arm and the same control program on four physics engines: PhysX, MuJoCo, MuJoCo Warp, and Genesis.
3D视觉工坊 · 微信公众号 ·
SenseTime and joint team evaluate GPT-6-Astra embodied capabilities, open-source SenseNova-RoboRSI framework and 27B model
A joint team including SenseTime and The Hong Kong University of Science and Technology (Guangzhou) released the report "Has GPT-6-Astra Achieved Embodied Intelligence?", systematically evaluating GPT-6-Astra's embodied capabilities across 12 benchmarks, and simultaneously open-sourced the SenseNova-RoboRSI evaluation framework and a 27B embodied model.
The Humanoid Hub ·
Ropedia releases open-world model W0, trained on about 10,000 hours of first-person human video
Ropedia released the open-world model W0, trained on about 10,000 hours of first-person human video. A single set of weights can predict the first-person view of humans, robotic arms, and humanoid robots (Unitree G1, Fourier GR-1), using a shared camera-plus-skeleton representation; adding a new robot requires only its URDF.
Xbotics具身智能实验室 · 微信公众号 ·
SenseTime launches SenseNova-RoboRSI and a 27B embodied model, achieving SOTA on 12 simulation benchmarks
The SenseTime SenseNova team launched the SenseNova-RoboRSI unified execution framework and the 27B embodied model SenseNova-RoboRSI-27B-Preview.
自变量机器人 · 微信公众号 ·
X Square Robot releases X2Real simulation benchmark, with 0.96 correlation between simulated and real-robot task success rates
X Square Robot has released X2Real, an evolvable simulation evaluation benchmark. Its simulated evaluation and real-robot evaluation achieve a task success rate correlation of 0.96 and a progress score correlation of 0.99.
arXiv cs.RO ·
Evaluating a Frozen Frontier VLM Directly as a Robot Policy on the RoboMME Memory-Dependent Manipulation Benchmark
The study uses a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy and tests it on RoboMME, a benchmark of 16 memory-dependent manipulation tasks. The model keeps the entire episode context throughout and directly outputs each end-effector waypoint and gripper command, with no training specific to this benchmark.
AI前线 · 微信公众号 ·
Google releases Gemini 4 Argon physical reasoning model, connecting the full robotics R&D chain
Google has released Gemini 4 (Argon), positioned as a general physical reasoning model focused on interaction with the physical world, covering the full robotics lifecycle from control execution and simulation reconstruction to data annotation and self-improvement. On the ERQA embodied reasoning benchmark, Argon scored 87%, surpassing Claude Opus 5.5 (82.3%) and GPT-6 Astra (81%); in the AA-Omniscience test, its hallucination rate was 15%, lower than the 51% and 59% of the latter two.
Xbotics具身智能实验室 · 微信公众号 ·
Shanghai AI Lab evaluates GPT-6-Astra's capability boundaries as a robot policy on EBench
Shanghai AI Laboratory's embodied evaluation team used EBench to evaluate GPT-6-Astra as a robot policy directly, without task-specific post-training, across 26 tasks and 510 total executions. Its average task success rate of 46.7% ranked third, and its average task score of 0.654 ranked second; OpenWAM-α led with a 55.3% success rate.
Dexmate ·
Dexmate and CMU Release MeMoREval Benchmark: Robot Storing Camera Frames Beats Text Descriptions
Dexmate and CMU introduced the MeMoREval benchmark, which uses a two-stage protocol of experiencing first and asking questions later to evaluate what embodied agents should store as memory. It includes 53 tasks and 4 simulated indoor environments, and runs on the Vega 1 Pro dual-arm mobile manipulator. Results show that storing raw camera frames scored 70% and had lower write cost, surpassing text descriptions at 56 to 60%; adding scene graphs achieved the highest overall success rate among all tested models. The protocol also completed two memory retrieval demonstrations on a real Vega 1 Pro using a SLAM map and head camera.