Event
AdaExploreBench Released to Evaluate Agents' Active Exploration Capabilities
1 reports · 1 sources
Researchers released the AdaExploreBench benchmark to evaluate the active exploration and test-time adaptation capabilities of LLM agents in evolving environments. The benchmark sets environment dynamics as hidden rules, requiring agents to actively acquire evidence to infer the rules; under implicit rule drift, rules change without notice, and agents need to detect the failure of old knowledge from performance degradation and re-explore. Experiments on eight leading LLMs show that active and adaptive exploration remains challenging: GPT-6 Sol's calibration accuracy is only 61.9%; providing sufficient evidence or removing rule drift can improve average performance by nearly 10%. The results indicate that agents can reason based on valid evidence, but struggle to actively acquire evidence and identify when evidence becomes invalid.
Report timeline
- AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-Time Adaptation in Evolving Environments
AdaExploreBench introduces a benchmark that instantiates environmental dynamics as hidden rules, requiring LLM agents to actively acquire evidence and infer the rules, and, under implicit rule drift, to detect that prior knowledge has become stale and re-explore. Experiments on eight leading LLMs show that GPT-6 Sol achieves only 61.9% calibrated accuracy; providing sufficient evidence or removing rule drift improves average performance by close to 10%. The results indicate that agents can reason from valid evidence but struggle to actively acquire evidence and to recognize when evidence becomes invalid.
HuggingFace Daily Papers ·