Event

physim-gate v0.1.1 switches to Newcombe intervals for 0/1 metrics

1 reports · 1 sources

physim-gate released v0.1.1, changing metrics that are entirely 0/1, such as success rate and collision indicator, from percentile bootstrap to Newcombe score intervals (hybrid score for unpaired cases, Newcombe paired method for paired trials), while also fixing the CLI exit code and input handling: directory input is now hashed instead of crashing with exit code 4. The project description states that bootstrap gives zero-width intervals in no-event runs, treating "0 collisions in 30 trials" as a ratio known to be zero, which could cause v0.1.0 to report "no regression" when the candidate is clearly worse; in simulations with 30 trials per run and a true collision rate rising from 1% to 6% (a 2 percentage point margin), v0.1.0 judged PASS in 15% of runs, while v0.1.1 gave 0%. The project also notes that with rare events and few trials, INCONCLUSIVE results will be more frequent.