Article

Semantic World Models: Using Vision-Language Models to Predict Semantic Information for Robot Planning

The paper proposes Semantic World Models, which model the world model as visual question answering about the semantic information of future frames rather than reconstructing future pixels, so that vision-language models can be obtained through supervised fine-tuning on image-action-text data. The authors validated it on simulated robot tasks, where it outperformed typical reconstruction-based action-conditioned world models.

Part of the event: Semantic World Models: Predicting Semantics with Vision-Language Models for Robot Planning

This page provides a summary and source link only.