MDF++ Visuotactile World Action Model Paper Released
What happened
Zixuan Huang and other authors released a paper proposing the visuotactile world action model MDF++: it uses a diffusion backbone network to end-to-end learn latent representations of multi-sensory observations, addressing the problem that existing world action models rely on frozen pretrained encoders and their representations are not suited for control. The authors state that its multi-task learning strategy leverages cross-modal correlations to learn task-relevant representations, avoiding the feature collapse caused by naive joint training. On real-world contact-intensive manipulation tasks of bolt tightening and nut alignment, the authors state that MDF++ outperforms world action models based on frozen pretrained representations by up to 50%.
AI 根据报道生成 · 2 days ago更新
Reporting timeline
Follow the reporting to explore different perspectives.
- arXiv cs.ROMDF++: A Visuo-Tactile World Action Model with End-to-End Representation Learning
MDF++ is a visuo-tactile world action model that uses a diffusion backbone to learn latent representations of multisensory observations end-to-end, addressing the problem that existing world action models rely on frozen pretrained encoders whose representations are not well suited for control. The authors propose a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations, avoiding the feature collapse caused by naive joint training. On contact-rich manipulation tasks including real-world bolt screwing and nut mating, it outperforms world action models built on frozen pretrained representations by up to 50%.
Activity for this event
- Current
- 3
- Peak
- 10Nov 9 14:00
- Past 24 hours
- -50%
The trend compares participants observed throughout the same period, so its range may be smaller than the current interest count. Move or tap to inspect each hour; use the arrow keys on a keyboard.