Event
Semantic World Models: Predicting Semantics with Vision-Language Models for Robot Planning
1 reports · 1 sources
Jacob Berg and co-authors propose Semantic World Models, which model the world model as visual question answering about the semantic information of future frames rather than reconstructing future pixels. The authors state that this perspective allows world models to reuse tools from vision-language models and enables training a vision-language model into a "semantic" world model via supervised fine-tuning on image-action-text data. The authors validate the method on simulated robot tasks and report that it outperforms typical reconstruction-based action-conditioned world models. This conclusion is currently limited to simulated tasks and does not yet involve real-robot experiments.
Report timeline
- Semantic World Models: Using Vision-Language Models to Predict Semantic Information for Robot Planning
The paper proposes Semantic World Models, which model the world model as visual question answering about the semantic information of future frames rather than reconstructing future pixels, so that vision-language models can be obtained through supervised fine-tuning on image-action-text data. The authors validated it on simulated robot tasks, where it outperformed typical reconstruction-based action-conditioned world models.
arXiv cs.RO ·