Skip to content
#Evaluations and benchmarks
Oct 24Sat
  1. 机器之心 · 微信公众号72

    Limpid Intelligence releases PrismWAM and RoboDoctor, topping RoboTwin 2.0 and RoboDojo leaderboards respectively

    Limpid Intelligence founder Li Yiming's team at Tsinghua University released the world action model PrismWAM and the execution diagnosis framework RoboDoctor. PrismWAM decouples semantic context from physical state and, without additional action-data pretraining, achieved an average success rate of 83.67% on RoboTwin 2.0 and 75.5% in the Clean-to-Random setting, ranking first on the official leaderboard.

  3. HuggingFace Daily Papers71

    RoboJev-27B: An Open-Source Vision-Language Model for Robot Manipulation Success Detection

    RoboJev-27B is an open-weight vision-language model that takes a task instruction and 10 frames of a manipulation attempt to judge whether the robot completed the task. It achieves 95.2% balanced accuracy and 0.993 AUROC on 268 held-out episodes of the same robot model (SO-101), and an AUROC of 0.890 on 459 episodes from unseen labs and robots. Median inference time is 0.26 seconds per episode on a single H100, and the model, inference code, and evaluation results for 29 model/protocol entries are released together.

  4. 简智机器人 · X60

    Gen-HumanEgo tops 550,000 HuggingFace downloads one month after release, ranking second among robotics datasets

    One month after release, GenRobot's open-source Gen-HumanEgo dataset has surpassed 550,000 downloads on HuggingFace, ranking second among robotics datasets. The dataset converts synchronized egocentric video via a Data Foundation Model into a unified token representation aligned across video, motion, hand tracking, depth, and semantic annotations, for robot learning and training. GenRobot says more open datasets will be released later.

Oct 23Fri
  1. 新智元 · 微信公众号58

    ZhiZhi Innovation Research Institute publishes IQuest-Q1 technical report: training details of a 320B open-source base model on one million tasks

    ZhiZhi Innovation Research Institute has published a complete technical report on IQuest-Q1, which has 320B total parameters and 15B activated parameters, disclosing results on 30 benchmarks and training details. The team built more than 1 million executable, containerized task instances from code repositories and existing agent trajectories, and set up a three-layer mechanism of environment preparation, runtime checks, and trajectory auditing to prevent reward hacking.

  2. arXiv cs.RO60

    EvSLAM: A Benchmark Framework for Event-Camera SLAM in High-Speed Maneuvers

    The authors propose EvSLAM, a benchmark framework for event-camera state estimation that covers multiple data-collection platforms, extreme lighting, and challenging motion patterns, and provides an explicit definition of high-speed maneuvers along with new evaluation metrics. The paper first analyzes the limitations of existing event-camera VO/VIO methods and the shortcomings of public datasets, then uses the framework to evaluate current state-of-the-art methods, summarizing better architectures and problems that remain to be solved.

  3. arXiv cs.RO62

    S²: Sensorimotor Alignment as an Offline Proxy for Embodied Policy Success

    The authors propose Sensorimotor Alignment (S²), a training-free offline metric that measures how well a model's sensory representations align with the geometric structure of physical motion. The study finds that S² correlates positively with downstream robot task success across multiple embodied model families, enabling low-cost model screening before closed-loop evaluation and identification of promising VLM backbones before fine-tuning into VLAs.

  4. arXiv cs.RO61

    RGraph-Eval: Evaluating Robot Video Generation with Task Scene Graphs

    The authors propose RGraph-Eval, a reference-demonstration-based framework for evaluating robot videos that uses task scene graphs to explicitly expose interaction errors in generated videos, such as errors involving objects, grasps, and placements. The framework uses a foundation-model pipeline to identify manipulation roles, locate object masks, infer and verify state transitions and action intervals, and reconstruct the 3D motion of manipulated objects. It outputs an interpretable score that covers visible objects, relation changes, and 3D path shape.

  5. arXiv cs.RO74

    LocoManiLab: A unified testbed for data, policies, controllers, and deployment in humanoid loco-manipulation

    LocoManiLab is a unified testbed for humanoid loco-manipulation covering data collection and generation, policy training, evaluation, and real-world deployment. Its task suite includes 234 atomic and compositional tasks across five household scene categories, supported by 51.5 hours of simulated teleoperation data.

  6. HuggingFace Daily Papers62

    S^2: Sensorimotor Alignment as an Offline Proxy for Embodied Policy Success

    The study proposes a training-free offline metric, Sensorimotor Alignment (S^2), that measures how well a model's sensory representations align with the geometry of physical motion. The authors report that S^2 correlates positively with downstream robot task success across model families, and that it can be used to screen models before closed-loop evaluation and to identify promising VLM backbones before costly VLA fine-tuning.

  7. 机器人全球资讯 · 微信公众号60

    Frost & Sullivan Report: Top Five Global Humanoid Robot Revenue Rankings for First Half of 2026 All Chinese Companies, UBTECH Leads

    International consulting firm Frost & Sullivan released the "2026 Global Humanoid Robot Market Research Report": in the first half of 2026, global humanoid robot sales reached 36,000 units and revenue reached USD 540 million, already exceeding the full-year 2025 figures of 35,000 units and USD 500 million.

  8. Robotics & Automation News72

    Google DeepMind uses Gemini 4 Argon to teach Apptronik's Apollo humanoid new skills from video

    Google DeepMind demonstrated a robot training technique alongside the release of Gemini 4 Argon: after watching raw video of a person playing cornhole, the model analyzes the environment, maps spatial coordinates, and computes full-body joint motion, automatically generating simulation code, then transfers the learned behavior to Apptronik's Apollo humanoid robot, which hit seven consecutive shots in real-machine testing.

  9. HuggingFace Daily Papers58

    FrontierCS 2: A Benchmark for Computer Science R&D Released

    Researchers introduce FrontierCS 2, a benchmark of 48 open-ended tasks. Their approach takes a paper and the code its authors released, deletes the paper's contribution, and uses the remaining code as the human reference baseline. Unlike earlier benchmarks that only give a problem description and grade one program after the run, FrontierCS 2 places agents inside a research environment where they inherit a runnable repository and can submit patches at any time. The evaluator returns only a development-suite score, while the evaluator, the held-out final suite, and the reference remain private.

Oct 22Thu
  1. Xbotics具身智能实验室 · 微信公众号77

    How to Choose a Robot Simulator: Same Success, Tenfold Contact Force Gap

    The article interprets a simulator comparison report released by Manda Robotics in early October, running about ten sets of experiments with the same Franka 7-axis robotic arm and the same control program on four physics engines: PhysX, MuJoCo, MuJoCo Warp, and Genesis.

    Why it matters: The article uses controlled comparison experiments under uniform conditions to show that simulator divergence is concentrated in contact forces rather than success or failure, and provides transferable troubleshooting and verification methods.

  2. 3D视觉工坊 · 微信公众号76

    SenseTime and joint team evaluate GPT-6-Astra embodied capabilities, open-source SenseNova-RoboRSI framework and 27B model

    A joint team including SenseTime and The Hong Kong University of Science and Technology (Guangzhou) released the report "Has GPT-6-Astra Achieved Embodied Intelligence?", systematically evaluating GPT-6-Astra's embodied capabilities across 12 benchmarks, and simultaneously open-sourced the SenseNova-RoboRSI evaluation framework and a 27B embodied model.

    Why it matters: The report breaks model capabilities into testable conditions and provides harness ablation data, so readers can judge the real bottlenecks in embodied capability evaluation and cost.

  3. 机器之心 · 微信公众号73

    RoboColiseum evaluates GPT, Claude, and Gemini robotics capabilities: Claude Opus 5.5 edges ahead, but general models still trail policy models

    Embodied intelligence evaluation platform RoboColiseum released an evaluation report: Claude Opus 5.5 ranked first among four multimodal models with 76.28 points, followed closely by GPT-6 Astra with 75.99 points, while GPT-6.1 Sol and Gemini 4 Argon scored 70.91 and 66.86 points respectively; the top two on the official overall leaderboard remain specialized embodied policy models Paimon (77.54 points) and HiDream-O1-Embodied (76.92 points).

  4. HuggingFace Daily Papers56

    SERA: A Self-Evolving Robotic Agent Framework for History-Dependent Manipulation

    The study proposes SERA (Self-Evolving Robotic Agent), which organizes a high-level VLM into a robotic agent through an editable harness, with a fixed VLA executor providing low-level control. The harness uses observations and execution feedback to maintain task context, and the self-evolution mechanism proposes and evaluates harness revisions through environment rollouts, with the selected configuration fixed before held-out testing. On RoboMME, SERA autonomously selects a configuration reaching 51.38% overall success rate, versus 44.38% without self-evolution.

  5. HuggingFace Daily Papers47

    PAPAV: A Unified Capability Framework for Multimodal Embodied Agents from Computer Use to Robot Use

    The paper proposes PAPAV, a capability-centric unified framework that characterizes multimodal embodied agents (MMEAs) through five functions: Perceive, Anticipate, Plan, Act, and Verify, and uses it to compare multimodal agents (MMAs) with robotic systems (RSs). An analysis across 64 benchmarks shows that explicit evaluation centers on Act, with only 5 assessing Anticipate and 3 assessing Verify. Physical constraints make errors harder to reverse, and PAPAV provides a common basis for designing reliable physical agents and measuring progress across the five capabilities.

  7. arXiv cs.RO55

    Hilti-Trimble-Oxford dataset released: a benchmark for 360-degree visual-inertial SLAM and floor plan localization

    The paper releases the Hilti-Trimble-Oxford dataset, collected at an active construction site. It contains 30 visual-inertial sequences covering 10 floors over an eight-month span, and captures real-world challenges such as changing lighting, moving workers, fast motion, and repetitive structures. Ground-truth trajectories were collected with a high-precision LiDAR-inertial SLAM system rigidly attached to the 360 camera. The authors also report results from the accompanying open research challenge: 62 teams entered the SLAM track, while only 22 entered floor plan localization, indicating less attention to the latter.

  10. arXiv cs.RO58

    TARSphere: A Robot-Oriented Full-Stack Framework and Benchmark for Long-Horizon Embodied Agents

    TARSphere is a robot-oriented full-stack framework for long-horizon embodied agents operating under continuously changing physical and social environments. It includes TARSphere-Infra, which runs robot policies in physics simulation; TARSphere-Harness, which lets a frozen multimodal large language model make decisions through multi-level executable skills; and TARSphere-Bench, an evaluation benchmark comprising 360 episodes across 30 indoor scenes.

  12. arXiv cs.RO51

    NavBind Diagnoses Instruction-Action Binding at Route Forks in Navigation VLA Models

    NavBind proposes a method for diagnosing whether navigation vision-language-action (VLA) models truly follow instructions: with the navigation context fixed, it checks whether changing the instruction affects the probability distribution of actions. The authors recombined route-aligned instruction segments from R2R-CE and RxR-CE to construct 3,252 route-fork contexts, each pairing two full instructions that share a description of the route already traveled but specify different branches, and evaluated them under the same agent state and navigation history.

  13. HuggingFace Daily Papers51

    SpatialBrain: A Spatial Intelligence Framework That Grows from Experience

    SpatialBrain is a framework that retains experience in spatial reasoning decisions on a fixed vision-language backbone. 3DFusionMemory fuses current-scene observations into queryable spatial relations, the Spatial Skill Library provides reusable tool procedures, and Spatial Cognition associates task conditions with experience about when and how to use memory and skills; completed trajectories are abstracted into process records and consolidated between task blocks.

  15. HuggingFace Daily Papers78

    Mechanistic Study of VLA Models: Not All Features Are Created Equal

    The study proposes an interpretability pipeline and finds that injecting task A's activations into the action expert causes pi0.5 to execute task A's motor trajectory in 99.6% of episodes (n=1,968), and that the injected program is bound to absolute workspace coordinates while ignoring the visible scene, offering a mechanistic explanation for the perturbation brittleness reported by benchmarks.

    Why it matters: The study uses intervention experiments to provide a mechanistic explanation for the brittleness of VLA policies, helping readers understand why models fail after slight scene changes.

  16. 未来机器人网 · 微信公众号61

    ACE Robotics releases embodied foundation model ACE-Brain-0.6, surpassing GPT-6 Astra on six of eight Grounding benchmarks

    ACE Robotics released ACE-Brain-0.6, a vision-spatial-anchoring-enhanced embodied foundation model focused on Grounding capability. It surpasses GPT-6 Astra on six of eight Grounding benchmarks, with an average score of 74.53 across seven public benchmarks (Astra: 73.68), and scores 94.13 on its self-built real-robot evaluation set RobotGrounding (2,591 questions, 14 task types; Astra: 91.10).

Oct 21Wed
  1. CVer · 微信公众号55

    NeurIPS 2026 Paper Urbex: Agent-Driven City-Scale 3D Spatial Localization

    Urbex is an agentic vision-language framework for city-scale 3D scenes that shifts from static ranking based on pre-generated candidate boxes to active spatial search, progressively narrowing the search area through three tools—landmark query, local zoom-in, and oblique view—and uses GRPO reinforcement learning to optimize tool-invocation strategies.

  3. arXiv cs.RO49

    TAPDreamer: Transferable Adversarial Patches for World Action Models

    TAPDreamer proposes an attack method that constructs a fixed local adversarial patch using only a public encoder, requiring no queries to the target policy and transferring across tasks and action architectures. In closed-loop evaluation, a frozen patch covering about 6.5% of the input reduced FastWAM's success rate from 97.7% to 0.0% on 40 LIBERO tasks and from 90.86% to 0.0% on 50 RoboTwin tasks, while a matched random patch still retained 81.5% and 79.2%.

  5. arXiv cs.RO56

    RoboJudge-Bench evaluates multimodal large models as judges of embodied video generation

    The paper introduces RoboJudge-Bench to assess the reliability of multimodal large language models (MLLMs) as judges of embodied video generation, covering two dimensions, physical consistency and instruction alignment, with 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands.

  6. arXiv cs.RO45

    Giving Models a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

    A study of 9 vision-language models, 682 images, and 7 evaluation conditions found that models' exact-counting ranges differ by nearly two orders of magnitude (R80 from 3 to 150 dots). Using a training-free agent loop that lets models mark images to aid counting, the study found this method mostly failed as an intervention but produced auditable serial reasoning traces that help reveal how models arrive at answers. Experiments also showed that models often change answers when traversal order is reversed, and that a child-like advantage from regular arrangements appeared only under specific conditions.

  7. arXiv cs.RO47

    When Does Experience Memory Actually Help Closed-Loop LLM Driving Agents? A Controlled Evaluation

    A controlled evaluation on the HighwayEnv testbed (1Hz, with gpt-4o-mini used for the key comparisons) shows that the published gains from experience memory in closed-loop LLM driving agents mainly come from sampling luck in the warmup pool: retrieval-based memory beat No-Memory in only one of ten pools (sign test p=0.011).

  8. arXiv cs.RO55

    TaskWAM uses stage-aware tokens to steer world-action models toward intended futures

    The paper proposes TaskWAM, a closed-loop, stage-aware framework that uses Mode Tokens to represent fine-grained task requirements and State Tokens to track execution progress, forming a structured task state that steers world-action model generation toward futures matching the current task, addressing the Stage Confusion phenomenon in long-horizon manipulation.

  9. arXiv cs.RO55

    CAPSafe benchmark evaluates safety reasoning for actions and long-horizon planning in embodied agents

    Researchers propose the CAPSafe benchmark, which evaluates the safety reasoning capabilities of large language model-driven embodied agents at two levels: an action-level dataset that judges whether a single action is safe in a given environmental context, and a plan-level dataset that assesses whether agents can generate safe long-horizon plans for complex hazardous household tasks.

  11. arXiv cs.RO62

    Reasoning driving VLAs have a perception-action binding gap: they can perceive danger but do not brake

    The paper points out that reasoning-augmented vision-language-action (VLA) driving models can perceive and reason about safety-critical hazards, yet cannot translate that evidence into safe actions, which the authors call a perception-action binding gap. Using matched simulated counterfactuals and logged replay, the study finds that in-path hazards are indeed encoded in the visual representation, but longitudinal planning is dominated mainly by ego-speed history and responds only weakly to hazard distance and the model's own reasoning trace.