Failure Boundary Profiles: A New Closed-Loop Evaluation Method

Milan Bulat and co-authors propose a "failure boundary profile" closed-loop evaluation method: each scenario family varies only one measurable factor (e.g., corridor width, oncoming traffic time gap), and each variant is labeled according to the measurement as either should-pass or should-stop for the agent. Along this axis, two curves are estimated for each agent—commitment (probability of attempting to pass) and capability (probability of passing without contact)—and their respective intersections with 0.5 are taken, yielding intervals that account for between-run variance of the simulator, with decision accuracy metrics obtained using certification labels. The authors evaluated seven driving agents on seven scenario families in CARLA and applied the method as-is to a warehouse robot in Isaac Sim. The authors report that, among the tested navigation policies, only one stops before blocked gaps in three scenario families; two fine-tuned versions pass more gaps more cleanly but no longer stop in any scenario family.

AI 根据报道生成 · 6 days ago更新