Event
Research on Developmentally Aligned Evaluation of VLM Counting Ability
1 reports · 1 sources
Yingxin Jia et al. propose a training-free evaluation method based on developmental psychology counting principles, auditing the counting ability of 9 vision-language models across 682 images and 7 evaluation conditions. Results show that the precise counting range varies by nearly two orders of magnitude across models (R80 from 3 to 150 dots), and is a model-specific characteristic. The study had models mark points on images to assist counting, and found that this method mostly fails as an intervention: marking helps only 4 models on small-count photos, harms models with imprecise pointing, and rarely outperforms verbal routines or blind dot lists. The value of this loop lies in leaving an auditable trace of serial reasoning rather than improving accuracy. Label perturbation experiments show that models with precise pointing read answers from labels, dense scenes fail due to omissions, and models rarely revise their answers after seeing their own labels.
Report timeline
- Giving Models a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models
A study of 9 vision-language models, 682 images, and 7 evaluation conditions found that models' exact-counting ranges differ by nearly two orders of magnitude (R80 from 3 to 150 dots). Using a training-free agent loop that lets models mark images to aid counting, the study found this method mostly failed as an intervention but produced auditable serial reasoning traces that help reveal how models arrive at answers. Experiments also showed that models often change answers when traversal order is reversed, and that a child-like advantage from regular arrangements appeared only under specific conditions.
arXiv cs.RO ·