Event
Video World Models Propose Predictable Latent Space Training Method
1 reports · 1 sources
Mattia Scardecchia and Sherry Yang propose a latent space training method for video world models, attempting to make the latent space both convenient for predicting the future and not at the expense of accurate pixel reconstruction; the specific approach is to add a temporal objective for representation learning into autoencoder training, together with a past-conditioned decoder architecture. The authors claim that under the same training budget, this method reduces inference cost by up to 5x and speeds up training by more than 7x, and reduces pixel and perceptual errors in out-of-distribution and long-horizon prediction by 15–24%; when a vision-language model judges task success based on pixel-space predictions, the world model's results are also closer to real policy evaluation. Based on this, the authors argue that temporal predictability should serve as a design principle for the latent space of video world models.
Report timeline
- Video World Models Need Predictable Latents
This paper proposes making the latent space of video world models balance predictability with pixel reconstruction accuracy, achieved through temporal objectives for representation learning paired with a past-conditioned decoder architecture. Under the same training budget, inference cost is reduced by up to 5x, training speed increases by more than 7x, and pixel and perceptual errors in out-of-distribution and long-horizon prediction drop by 15-24%. The authors argue that temporal predictability should be an important design principle for video world model latent spaces and provide a concrete method for incorporating it into autoencoder training.
arXiv cs.RO ·