Event
L2A Framework: Vision-Language Priors Plus Tactile Online Identification of Part Motion Patterns
1 reports · 1 sources
Mohamed A. Mabrok proposes the Open-World Law-to-Action (L2A) framework: a vision-language model proposes, for each movable part, its motion type, motion-law terms, and possible locking factors; Bayesian linear regression with closed-form evidence is then used to online-identify the part's law, and touch can overturn the prior. All experiments with this method are currently conducted in simulation, developed on some scenes and run once each on held-out scenes. In a new kitchen, the robot completes 80% of tasks following the vision-language model's proposals, versus 98% when all parts are known; a mobile robot that takes no information from the simulator completes 70 of 100 requests in a new scene, versus 62 when the simulator tells it which part corresponds to each box (one change was made between the two held-out tests).
Report timeline
- Open-World Law-to-Action: Using vision-language priors and touch to infer the motion laws of parts with hidden physics
The authors propose the Open-World Law-to-Action (L2A) framework, in which a vision-language model proposes, for each movable part, its motion type, the terms of its law of motion, and possible locking factors; the part's law is then identified online via Bayesian linear regression with closed-form evidence, and touch can overturn the priors.
arXiv cs.RO ·