Event
CAPSafe Benchmark Evaluates Safety Reasoning Capabilities of Embodied Agents
1 reports · 1 sources
Youmna Farag and other researchers have released the CAPSafe benchmark, which is used to evaluate the safety reasoning capabilities of embodied agents driven by large language models, covering both the action level and the planning level. After evaluating nine closed-source and open-source large models, they found a clear gap between the models' hazard identification and safe long-horizon planning; Gemini-3-pro had the highest safety success rate, at 78.19%, while all other models were below 65%. In the CAPSafe setup, hazards are contextual (requiring inference from object attributes), multiple hazards can exist simultaneously within the same environment, and each hazard is annotated with a relative severity level to enable fine-grained analysis.
Report timeline
- CAPSafe benchmark evaluates safety reasoning for actions and long-horizon planning in embodied agents
Researchers propose the CAPSafe benchmark, which evaluates the safety reasoning capabilities of large language model-driven embodied agents at two levels: an action-level dataset that judges whether a single action is safe in a given environmental context, and a plan-level dataset that assesses whether agents can generate safe long-horizon plans for complex hazardous household tasks.
arXiv cs.RO ·