MDF++: A Visuo-Tactile World Action Model with End-to-End Representation Learning

MDF++ is a visuo-tactile world action model that uses a diffusion backbone to learn latent representations of multisensory observations end-to-end, addressing the problem that existing world action models rely on frozen pretrained encoders whose representations are not well suited for control. The authors propose a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations, avoiding the feature collapse caused by naive joint training. On contact-rich manipulation tasks including real-world bolt screwing and nut mating, it outperforms world action models built on frozen pretrained representations by up to 50%.