Event

Controlled evaluation: experiential memory gains mostly come from warm-up pool sampling luck

1 reports · 1 sources

Yingxin Jia and Xuanyi Li conducted a controlled evaluation of experiential memory in closed-loop LLM driving agents on an audited HighwayEnv testbed (1 Hz, with gpt-4o-mini used for key comparisons). The results show that published memory gains mainly come from sampling luck in the warm-up pool: after treating the warm-up pool as a resamplable random factor (ten mutually exclusive pools), retrieval-based memory beat No-Memory on only one pool (sign test p=0.011). The evaluation covered five types of memory schemes (DiLu-style flat text retrieval, RESPOND-style single-frame grid transfer, episodic temporal graph memory, retrieval-free rulebook, and a K-step observation history baseline), and added a token-matched irrelevant filler control and a baseline transfer audit. The authors state that all four findings are properties of this testbed, and they released the testbed, audited transfers, seeds, and caches.