Skip to content
#Evaluations and benchmarks
Oct 21Wed
  1. arXiv cs.RO55

    EventSync: Event-driven memory updates for history-dependent robotic manipulation tasks

    EventSync proposes a hierarchical framework with event-driven memory updates to address history-dependent manipulation tasks where the current observation alone is insufficient to determine the next action. A vision-language model generates subgoals based on the task plan and interaction history, while a vision-language-action model handles execution. A lightweight Event Monitor combines visual observations with end-effector trajectories to determine whether the current subgoal is complete; only after obtaining completion evidence does it write the completed subgoal into the history and trigger subsequent reasoning.

  3. arXiv cs.RO57

    RoboCasa-Shift: A Stage-Aware Robustness Evaluation Benchmark for VLA Policies in Long-Horizon Household Manipulation

    The paper proposes RoboCasa-Shift, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor perturbations, paired clean-perturbed rollouts, and simulator-grounded stage-aware diagnostics, completing a total of 1,040 rollouts evaluating GR00T N1.5 across four long-horizon tasks, four perturbation factors, and three severity levels.

  5. arXiv cs.RO48

    Evaluating Social Reasoning of Gemini Robotics-ER on the SHREC Benchmark

    An evaluation tested the social reasoning capabilities of Gemini Robotics-ER on the SHREC human-robot social interaction benchmark and compared it with published general-purpose vision-language model baselines. Results show that Gemini Robotics-ER outperforms GPT-4o-mini on error/competence detection but performs worse on fine-grained social-attribute judgments; adding a visual frame helped one task while hurting the other.

  6. arXiv cs.RO55

    HATCH: A Hardware-Attested Task-Chain Benchmark for Verifiable Manipulation

    HATCH is an open-specification, physically replicable, sensor-scored, and unrehearsable robot manipulation benchmark, accepted to the 5th Open Robot Software Workshop at IROS 2026 (2 pages, 1 table). Tasks span five skill categories and three difficulty levels, and a sensor that confirms completion of one task unlocks the next, chaining an entire room into a single long-horizon evaluation.

  8. arXiv cs.RO44

    Research on Interaction State Recognition for Gecko-Inspired Adhesive Interfaces for On-Orbit Servicing

    A study embeds capacitive sensing into a gecko-inspired dry adhesive interface to recognize five interaction states in real time during on-orbit servicing grasps (idle, proximity, contact, engaged, disaged). Random Forest achieved 95.65% offline classification accuracy; online, combined with a rule-based state machine, accuracy rose from 78.02% to 93.14%, with a macro F1 of 0.9343 and an inference cycle of 53 ms.

  9. arXiv cs.RO56

    ACE-Brain-0.6 Released: Scaling Spatial Perception of a 9B Embodied Model with Verifiable Grounding Supervision

    The research team released ACE-Brain-0.6, which uses a unified data pipeline to combine robot manipulation recordings with complementary visual data from multiple sources, automatically annotates them, and converts the result into verifiable grounding supervision; the model was obtained by fine-tuning Qwen3.5-9B on approximately 1.5 million samples. ACE-Brain-0.6 outperforms GPT-6 Astra on four of seven public grounding benchmarks, with an overall average score of 75.41 versus 73.68; under a fixed fine-tuning setup, overall grounding performance improves steadily as the dataset expands to cover more grounding scenarios.

  10. arXiv cs.RO63

    Embodied Playground: A benchmark for physical, strategic, and spatial reasoning in long-horizon agentic robots

    The authors propose Embodied Playground, a benchmark for evaluating how foundation-model agents coordinate physical, strategic, and spatial reasoning across long-horizon tasks. It contains 410 task environments spanning basic manipulation, sequential puzzles, and multi-stage household activities.

  13. OFweek Robotics65

    Geek+ Gravity Control uses natural language for runtime correction, raising OOD task success rate from 5% to 95%

    Under the Gravity framework, Geek+ embodied intelligence proposes the Gravity Control causal intervention model. Without training the original policy model or adding new demonstration data, it uses mechanistic interpretability to uncover the model's internal action regulation mechanisms and intervenes on relevant latent features at inference time to achieve runtime correction.

  14. 钛媒体34

    Edge AI Daily Morning Brief (October 21)

    An evaluation by SenseTime together with teams including Daxiao Robotics and Nanyang Technological University shows that GPT-6-Astra came close to full marks on several of 12 simulation benchmarks (100% on LIBERO-Long++, 98.8% on SIMPLE), but after deployment on 8 real robot platforms, all 11 attempts to plug in a plug failed and folding clothes ran into trouble. Its high scores also depend on two "boosters": in-context learning (ICL) and relaxing the action step limit to 10x; on the long-horizon household benchmark BEHAVIOR, its success rate was only 37.0%.

  15. HuggingFace Daily Papers58

    TARSphere: A Full-Stack Framework and Benchmark for Long-Horizon Embodied Agents

    TARSphere proposes a full-stack framework for robots that evaluates long-horizon embodied agents under physical and social constraints. Its TARSphere-Infra runs robot policies in a physics simulator with embodied NPCs and constrained dialogue, and uses an event-driven finite-state machine to update task states based on completed actions and delivered information; TARSphere-Harness lets a frozen multimodal large language model revise its decisions through multi-level executable skills, observations, hierarchical memory, and execution feedback.

Oct 20Tue
  1. The Humanoid Hub76

    Ropedia releases open-world model W0, trained on about 10,000 hours of first-person human video

    Ropedia released the open-world model W0, trained on about 10,000 hours of first-person human video. A single set of weights can predict the first-person view of humans, robotic arms, and humanoid robots (Unitree G1, Fourier GR-1), using a shared camera-plus-skeleton representation; adding a new robot requires only its URDF.

    Why it matters: The original article provides the training data scale, the cross-embodiment transfer method, and measured comparisons, so readers can judge the world model's actual impact on robot policy training workflows.

  2. ModelScope Robotics69

    Xingchen releases and open-sources intent understanding model Xing4.0-29B-Intent

    The Xingchen General Artificial Intelligence Laboratory of China Telecom Artificial Intelligence Technology Co., Ltd. has released and open-sourced Xing4.0-29B-Intent, a model dedicated to intent understanding, and has simultaneously opened 2.753 million supervised fine-tuning training corpus entries, model weights, and inference images, with the code licensed under Apache 2.0.

  4. 深蓝AI · 微信公众号72

    Zhejiang University's Gao Fei team releases CrossFly benchmark: evaluation of cross-indoor-outdoor interactive navigation for drones and a hierarchical VLA baseline

    The Gao Fei team at Zhejiang University, together with Weifen Zhifei, launched the CrossFly benchmark, building 23 reconstructed real-world scenes and 15,263 complete trajectories, covering three task types: indoor, outdoor, and indoor-outdoor cross-domain, accompanied by a hierarchical VLA baseline, CrossFly-Agent; the full simulation dataset, evaluation protocol, and baseline code are all open source.

  5. HuggingFace Daily Papers56

    RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models

    RoboValue introduces a fine-grained sim-and-real benchmark for unified evaluation of robotic value models, assessing four complementary capability dimensions through shared interfaces: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency.

  6. HuggingFace Daily Papers55

    PhysRSI-eval: An Observational Scaling Law Study of Physical Recursive Self-Improvement

    This paper systematically studies the mechanisms, scaling behavior, and generalization of physical recursive self-improvement (RSI) in embodied agents, proposing PhysRSI-eval, which quantifies physical RSI across five dimensions: first-attempt performance, efficiency, success rate, budget, and improvement from failure, and evaluates multiple VLMs on 18 manipulation tasks under a fixed interaction budget.

  7. HuggingFace Daily Papers56

    HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

    The research team proposes HandsOn, a composable cross-embodiment dexterous manipulation simulation platform and benchmark supporting 5 robotic arms, 6 dexterous hands, and about 200 rigid, articulated, and deformable assets. The benchmark contains 18,000 human demonstrations and 66 human-verified tasks, of which Core-50 is used for demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task demonstrations.

  8. HuggingFace Daily Papers50

    AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-Time Adaptation in Evolving Environments

    AdaExploreBench introduces a benchmark that instantiates environmental dynamics as hidden rules, requiring LLM agents to actively acquire evidence and infer the rules, and, under implicit rule drift, to detect that prior knowledge has become stale and re-explore. Experiments on eight leading LLMs show that GPT-6 Sol achieves only 61.9% calibrated accuracy; providing sufficient evidence or removing rule drift improves average performance by close to 10%. The results indicate that agents can reason from valid evidence but struggle to actively acquire evidence and to recognize when evidence becomes invalid.

  9. HuggingFace Daily Papers62

    AnyGoal: Evaluating World Models' Compositional Generalization Beyond Expert Trajectories

    The study introduces the AnyGoal benchmark and open-source library to evaluate world models' compositional generalization on goal-shifted task variants. The task design ensures that individual state factors and actions remain within the training support, but their combinations are underrepresented in expert trajectories. Experiments across multiple environments and world model families show that current models consistently fail under these recombination scenarios, and the authors further analyze where failures occur at the dynamics and representation levels.

  10. HuggingFace Daily Papers56

    Stateful Harness: Separating World State Evolution from Video Rendering to Support Rule-Governed Interaction

    The paper proposes the Stateful Harness framework, which separates world evolution from video rendering, using a frozen video model to render events and write back the world state, in order to support rule-governed interaction with hidden state. The authors build the WorldState-Bench benchmark, containing 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome.

  11. 雷科技73

    X Square Robot releases X2Real simulation benchmark, with 0.96 correlation between simulated and real-robot task success rates

    X Square Robot released the X2Real simulation benchmark for evaluating the performance of embodied models in the real physical world, proposing three design principles: credibility, validity, and objectivity. In testing, X2Real simulation evaluation achieved a 0.96 correlation with real-robot evaluation in task success rate and a 0.99 correlation in progress score; control parameter calibration achieved millimeter-level trajectory consistency, covering embodiments including Franka, ArtiXon Arm-6A, ARX R5, and Quanta X1.

  12. PaperWeekly · 微信公众号74

    SenseTime SenseNova-RoboRSI unified Harness raises GPT-6-Astra success rate on RoboDojo from 15.83% to 54.17%

    A joint research team from SenseTime, Nanyang Technological University and other institutions has introduced the SenseNova-RoboRSI unified Harness, which raises GPT-6-Astra's manipulation success rate on 42 RoboDojo tasks from 15.83% in the baseline configuration to 54.17%, with model parameters unchanged.

  15. Tech Xplore Robotics62

    Seoul National University's Dynin-Robotics robot foundation model guides actions by predicting future scenes, achieving 78.4% average success rate in real-world operation

    Professor He Jiong's team at Seoul National University's AIDAS Lab proposed Dynin-Robotics, a robot foundation model that unifies language, images, video, and robot actions in a single model, generating and selecting actions by predicting future scene images after an action; the paper has been accepted by NeurIPS 2026.

Oct 19Mon
  1. CVHub · 微信公众号74

    XPeng and multiple universities propose 4B perception foundation model GroundingPI, unifying point, box, and text localization and validating action transfer

    Researchers from XPeng and Peking University, the University of Hong Kong, UC Berkeley, Princeton University, National University of Singapore, Tsinghua University, and Hong Kong University of Science and Technology (Guangzhou) jointly proposed a 4B perception foundation model, GroundingPI, which unifies point selection, boxes, text, document regions, and interface controls into a single spatial token output protocol. The architecture uses a MoonViT-V2 vision encoder and a Qwen3-4B language backbone, and training includes supervised fine-tuning and GRPO reinforcement post-training.

  2. 机器人前瞻 · 微信公众号73

    Ropedia and multiple universities release cross-embodiment mobile manipulation world model Ropedia W0

    Singapore-based embodied intelligence startup Ropedia, together with Nanyang Technological University, Duke University, UC San Diego, and Tsinghua University, released the cross-embodiment mobile manipulation world model Ropedia W0, which uses a unified action representation to predict first-person future scenes after humans, robotic arms, and humanoid robots perform actions.

  3. Xbotics具身智能实验室 · 微信公众号76

    SenseTime launches SenseNova-RoboRSI and a 27B embodied model, achieving SOTA on 12 simulation benchmarks

    The SenseTime SenseNova team launched the SenseNova-RoboRSI unified execution framework and the 27B embodied model SenseNova-RoboRSI-27B-Preview.

    Why it matters: Through comparison data from the same model with different execution systems, readers can intuitively understand the magnitude of the execution framework's impact on embodied task performance and its cost.

  4. 自变量机器人 · 微信公众号74

    X Square Robot releases X2Real simulation benchmark, with 0.96 correlation between simulated and real-robot task success rates

    X Square Robot has released X2Real, an evolvable simulation evaluation benchmark. Its simulated evaluation and real-robot evaluation achieve a task success rate correlation of 0.96 and a progress score correlation of 0.99.

    Why it matters: It provides correlation data between simulated and real-robot evaluation along with a unified integration method, allowing readers to judge whether simulated evaluation can replace part of real-robot testing.

  7. 深蓝AI · 微信公众号62

    Daxiao Robotics releases ACE-Brain-0.6: 9B model beats GPT-6 Astra on six Grounding benchmarks

    Daxiao Robotics released the embodied spatial perception model ACE-Brain-0.6, built on Qwen3.5-9B with a frozen vision encoder and supervised fine-tuning of the language model parameters. It surpasses GPT-6 Astra on six of eight Grounding benchmarks, leads with an average of 74.53 points across seven public benchmarks versus Astra's 73.68, and scores 94.13 on RobotGrounding versus 91.10.

  9. 36Kr Embodied Intelligence36

    MetaDecision enters JevBench open-source top three: 31B version has median single-decision latency under 0.2 seconds

    MetaDecision, the structured decision model from MetaDecision Lab, has entered the top three on the JevBench decision model evaluation leaderboard: as of October 17, among 129 open-source systems, its 31B version ranks 3rd, and its 12B version ranks first among models of the same size. According to the leaderboard's GPU measurements, the 31B version has a median single-decision latency of under 0.2 seconds (excluding network). MetaDecision is an AI lab focused on Agent Infra and decision models, offering an embeddable Agent runtime foundation as well as knowledge and model services.

  10. arXiv cs.RO64

    GroundingPI releases 4B grounding foundation model, averaging 73.68% across 34 benchmarks

    GroundingPI is a 4B-parameter grounding foundation model that outputs points and boxes as quantized coordinates in a shared vocabulary; its training combines multimodal and spatial pretraining, supervised fine-tuning, and GRPO reinforcement learning. It averages 73.68% across 34 grounding benchmarks and 44 baselines, above GPT-6 Astra's 71.54%.