Event
Dexmate and CMU Release MeMoREval Memory Benchmark
1 reports · 1 sources
Dexmate and Carnegie Mellon University released the MeMoREval benchmark, investigating what form of memory embodied agents should retain. Results show that storing raw camera frames scored 70% and had lower write cost, exceeding text descriptions at 56% to 60%; adding a scene graph yielded the highest overall success rate across all tested models. The benchmark uses a two-stage protocol of experiencing first and questioning later: the agent first explores the environment to accumulate memories, then in the request stage answers or acts solely from memory and cannot re-observe. The evaluation includes 53 tasks, covering four simulated indoor environments—an office, a photography studio, a residence, and a clinic—each with 7 to 9 rooms and hundreds of objects; the simulated robot is the Vega 1 Pro dual-arm mobile manipulator. The team also conducted two memory retrieval demonstrations on a real Vega 1 Pro using a SLAM map and a head camera, but stated that this is only a feasibility validation, not the research.
Report timeline
- Dexmate and CMU Release MeMoREval Benchmark: Robot Storing Camera Frames Beats Text Descriptions
Dexmate and CMU introduced the MeMoREval benchmark, which uses a two-stage protocol of experiencing first and asking questions later to evaluate what embodied agents should store as memory. It includes 53 tasks and 4 simulated indoor environments, and runs on the Vega 1 Pro dual-arm mobile manipulator. Results show that storing raw camera frames scored 70% and had lower write cost, surpassing text descriptions at 56 to 60%; adding scene graphs achieved the highest overall success rate among all tested models. The protocol also completed two memory retrieval demonstrations on a real Vega 1 Pro using a SLAM map and head camera.
Dexmate ·