Skip to content
Only Research#Evaluations and benchmarks
  1. arXiv cs.RO80

    Paper analyzes agentic manipulation policies: AMP1 achieves 85.6% zero-shot success rate

    The paper proposes agentic manipulation policies (AMP), in which a generalist AI model directly controls a robotic arm and adjusts its actions as the task progresses. The authors' reference implementation, AMP1, runs unchanged across single-arm, bimanual, and mobile manipulation.

    Why it matters: The paper reports zero-shot success rates and real-world trial data for a generalist AI model directly controlling a robotic arm, which can be compared with specialist policies such as VLA and WAM.

  1. 机器之心 · 微信公众号78

    Northwestern, CMU and others propose WOVEN: cross-domain gains on 22 benchmarks with 2k samples, plus a training recipe

    A team from Northwestern, CMU, UNC Chapel Hill and Amazon, in the paper "WOVEN: Weaving Visual World Modeling into Multimodal LLMs," defines world modeling as estimating the state transition p(s′|s,a) and builds WOVEN, a training and evaluation base of 36K questions.

    Why it matters: The paper breaks world modeling into independently intervenable state-transition tasks and provides reusable rules for selecting training data, offering reference value for multimodal model training.

  1. HuggingFace Daily Papers85

    PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations

    The research team proposes PrimeSim, an action-conditioned multiview world model simulator, midtrained on robot demonstration data covering 22 platforms and 30,110 hours, and has released the model weights, inference code, and training code.

    Why it matters: This work presents a world model simulator trained on demonstrations spanning 22 platforms and 30,000 hours, and releases the weights and training code, facilitating comparison of its long-horizon stability and action-following capability.

  1. Xbotics具身智能实验室 · 微信公众号78

    GE-Sim 2.0 and π0.5 closed-loop simulation test: Why does reward hacking occur in VLA training?

    The Xbotics Embodied Intelligence Lab built a closed-loop simulation pipeline using the AgiBot GE-Sim 2.0 world model and the OpenPI π0.5 policy. In testing, the world model generated about 4.33 frames per second, the pure-policy closed loop ran at about 3.20 frames per second, and after adding MiMo online scoring, the rate dropped to about 0.88 frames per second.

    Why it matters: The author uses a four-stage test to break down the frame-rate bottlenecks and causes of reward hacking in world-model closed-loop simulation, which can be transferred to the design of VLA evaluation workflows.

  1. arXiv cs.RO80

    SimuGuard: Auditing Contact Artifacts in Simulation Benchmarks Such as LIBERO and RoboTwin

    The authors propose SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa, and RoboDojo that identifies failures caused by the simulator rather than the policy. On RoboTwin, the authors trace these failures to the physics engine's collision detection: when geometry is 1 to 2 centimeters apart, the engine reports a penetration of about 2 centimeters, and the solver ejects the object at several meters per second because the distance algorithm discards triangles below a fixed area threshold.

    Why it matters: The original traces failures in simulation benchmarks to a physics engine collision detection constant and provides a before-and-after success rate comparison, which can serve as a reference for teams doing simulation evaluation.

  2. arXiv cs.RO77

    From Teleoperation to Long-Horizon Dexterity: Evaluating Visuomotor Skills and Offline Proxies for Dexterous Manipulation

    The research team used a five-fingered robotic hand to grasp and insert RJ45 connectors, evaluating 281 imitation-learning policies spanning seven policy families in approximately 1,700 real-robot trials. Controlled ablations showed that observation and action representations had a greater impact than conventional optimizer and sampling choices. For point-cloud policies, region-of-interest cropping, representation in the robot-base frame, and normalization within the point encoder each improved success by 70 to 100 percentage points. For the vision-language-action policy, switching to absolute action prediction improved insertion success by up to 70 percentage points.

    Why it matters: Approximately 1,700 real-robot trials compared 281 imitation-learning policies, indicating that observation and action representations determine real-world success more than optimizer and sampling choices.

  3. HuggingFace Daily Papers76

    RoboColosseum: A distributed real-world robot policy evaluation framework across four embodiments

    RoboColosseum is a distributed evaluation framework that evaluates general-purpose robot policies over a network of locally operated robots in multiple locations, using unified policy serving and standardized protocols.

    Why it matters: This distributed evaluation framework used 5,699 paired trials to compare VLA policies with frontier agents, revealing inconsistent rankings across embodiments and a ranking reversal after fine-tuning.

  1. HuggingFace Daily Papers78

    Mechanistic Study of VLA Models: Not All Features Are Created Equal

    The study proposes an interpretability pipeline and finds that injecting task A's activations into the action expert causes pi0.5 to execute task A's motor trajectory in 99.6% of episodes (n=1,968), and that the injected program is bound to absolute workspace coordinates while ignoring the visible scene, offering a mechanistic explanation for the perturbation brittleness reported by benchmarks.

    Why it matters: The study uses intervention experiments to provide a mechanistic explanation for the brittleness of VLA policies, helping readers understand why models fail after slight scene changes.

  1. 机器之心 · 微信公众号76

    Tsinghua team builds HandsOn dexterous manipulation benchmark with 66 tasks to evaluate robot bimanual capabilities

    Ding Ning's team at Tsinghua University, together with researchers from Beijing University of Posts and Telecommunications, Peking University, and Nature Will, built HandsOn, a simulation platform and evaluation benchmark for cross-embodiment dexterous manipulation, containing 66 manually verified tasks and 18,000 human simulation teleoperation demonstrations.

    Why it matters: Readers can learn why dexterous manipulation evaluation should distinguish execution from transfer, and which specific failure points current multi-task policies have on bimanual collaboration tasks.

  1. 自变量机器人 · 微信公众号74

    X Square Robot releases X2Real simulation benchmark, with 0.96 correlation between simulated and real-robot task success rates

    X Square Robot has released X2Real, an evolvable simulation evaluation benchmark. Its simulated evaluation and real-robot evaluation achieve a task success rate correlation of 0.96 and a progress score correlation of 0.99.

    Why it matters: It provides correlation data between simulated and real-robot evaluation along with a unified integration method, allowing readers to judge whether simulated evaluation can replace part of real-robot testing.

  2. arXiv cs.RO78

    Evaluating a Frozen Frontier VLM Directly as a Robot Policy on the RoboMME Memory-Dependent Manipulation Benchmark

    The study uses a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy and tests it on RoboMME, a benchmark of 16 memory-dependent manipulation tasks. The model keeps the entire episode context throughout and directly outputs each end-effector waypoint and gripper command, with no training specific to this benchmark.

    Why it matters: The paper provides complete comparative data on a frozen frontier VLM used directly as a policy on a memory-dependent manipulation benchmark, which can be used to assess how much of the embodied stack a general-purpose model can take over.

  3. 3D视觉工坊 · 微信公众号76

    SenseTime and joint team evaluate GPT-6-Astra embodied capabilities, open-source SenseNova-RoboRSI framework and 27B model

    A joint team including SenseTime and The Hong Kong University of Science and Technology (Guangzhou) released the report "Has GPT-6-Astra Achieved Embodied Intelligence?", systematically evaluating GPT-6-Astra's embodied capabilities across 12 benchmarks, and simultaneously open-sourced the SenseNova-RoboRSI evaluation framework and a 27B embodied model.

    Why it matters: The report breaks model capabilities into testable conditions and provides harness ablation data, so readers can judge the real bottlenecks in embodied capability evaluation and cost.

  1. The Robot Report78

    WorldOracle releases action-conditioned world model benchmark; seven models show deviations between predictions and real execution

    Princeton University, UT Austin, the University of Toronto, NVIDIA, and AgiBot jointly launched WorldOracle, an action-conditioned world model benchmark that uses the same initial frame and action sequence to compare model predictions with real execution.

    Why it matters: The benchmark compares world model predictions frame by frame against real execution of the same actions and releases failure samples, making it easier to identify sources of model deviation over long sequences.

  1. Dexmate72

    Dexmate and CMU Release MeMoREval Benchmark: Robot Storing Camera Frames Beats Text Descriptions

    Dexmate and CMU introduced the MeMoREval benchmark, which uses a two-stage protocol of experiencing first and asking questions later to evaluate what embodied agents should store as memory. It includes 53 tasks and 4 simulated indoor environments, and runs on the Vega 1 Pro dual-arm mobile manipulator. Results show that storing raw camera frames scored 70% and had lower write cost, surpassing text descriptions at 56 to 60%; adding scene graphs achieved the highest overall success rate among all tested models. The protocol also completed two memory retrieval demonstrations on a real Vega 1 Pro using a SLAM map and head camera.

    Why it matters: The benchmark was proposed jointly by the platform developer and CMU; its frame-versus-scene-graph comparison can help with memory design choices, and the real-robot transfer offers a feasibility reference.

  1. 智源研究院 · 微信公众号75

    BAAI releases FlagEval-Robo embodied intelligence evaluation system and dual leaderboards

    BAAI released the FlagEval-Robo embodied intelligence evaluation system on August 21, launching the FlagEval-Robo-Sim and FlagEval-Robo-Real leaderboards and publishing evaluation results for 12 open-source models. All models underwent equal post-training on the evaluation dataset.

    Why it matters: The evaluation uses a unified dual-track benchmark to quantify the performance gap between simulation and real robots, helping readers judge the limits of how far simulation leaderboard results can be generalized.

  1. PKU EPIC Lab66

    PKU EPIC Lab releases HumanTracker humanoid motion tracking benchmark and HumanScore metric

    PKU EPIC Lab proposed the HumanTracker benchmark for evaluating humanoid motion tracking, accepted to ECCV 2026. The benchmark contains approximately 153 hours of optical motion trajectories from multiple professional performers, organized into four motion families with text labels; the accompanying HumanScore metric is trained on 12K motion pairs containing 24K motions, aligns with human preferences, and can reveal contact and stability failures that kinematic metrics often miss, such as foot skating and mistimed landings.

    Why it matters: The original work proposes a benchmark and metric that align humanoid motion tracking evaluation with human perception, and points out that existing kinematic errors miss contact issues such as foot skating.

  1. Robot.com Holdings Inc.68

    Robot.com releases R-KNav 10,000-hour sidewalk navigation dataset and launches FoMo Challenge

    Robot.com has released the R-KNav dataset and launched the FoMo Challenge, inviting the community to train visual navigation models on real data from its R-kiwi sidewalk delivery robot fleet and to evaluate them in controlled tests on physical robots.

    Why it matters: The data comes from a long-running commercial delivery fleet rather than simulation, and the KPI criteria for physical-robot evaluation have been published, making it possible to assess the reusability of real-street navigation data.

  1. Berkeley AUTOLAB68

    CaP-X: Benchmarking and Improvement Framework for Robot Manipulation Coding Agents Released

    Teams including Berkeley AUTOLAB propose CaP-X, an open framework for systematically studying Code-as-Policy agents in robot manipulation, centered on the interactive environment CaP-Gym and CaP-Bench, which evaluates 12 frontier models.

    Why it matters: The original text presents a systematic evaluation framework for Code-as-Policy agents and the finding that test-time scaling can mitigate abstraction dependence, offering reference value for research on coding-agent manipulation control.

  1. Dexterity60

    Dexterity details evaluation methods and production metrics for the Foresight packing agent

    Dexterity published an article introducing system evaluation and production measurement for the Foresight packing agent. On the planning side, it was tested on four real warehouse box-type distributions, with density of 0.65–0.77 and stability of 99.7–100%, and distribution-free order statistics were used to give 95% confidence intervals; the article also found that randomly shuffling the box sequence causes local variance to collapse by about 1.4 times, making evaluation overly optimistic.

    Why it matters: The original article fully discloses the evaluation method for the packing agent, including distribution-free confidence intervals and the evaluation bias caused by random shuffling; the method can be transferred to similar systems.

You’re all caught up