Failure-Boundary Profiles: A New Method for Closed-Loop Evaluation of Driving and Robot Navigation Agents

The study proposes a failure-boundary profile evaluation method in which each scenario family varies only one measurable factor (such as corridor width or the time gap in oncoming traffic), and each variant is labeled from measurement as one where the agent should proceed or one where it should stop. Along this axis, two curves are estimated for each agent: commitment (the probability of attempting the passage) and competence (the probability of passing without contact). The method locates where each curve crosses 0.5 and gives intervals that account for the simulator's run-to-run variance, while also using certified labels to derive a decision accuracy metric.