FRED self-distillation framework improves robot future video representations
What happened
Huiwon Jang and other authors propose the FRED framework, which trains a world action model (WAM) to predict future video representations from noise in a single denoising step. The approach uses representations extracted from real future observations as the self-distillation target and uses historical observations to provide temporal context for the video model. The authors report that in a variety of real manipulation scenarios, FRED outperforms the compared WAMs, with notable improvements on long-horizon and out-of-domain tasks under zero-shot environment generalization; on tasks that are absent from the robot training data and demonstrated only by human video, FRED achieves a 50.0% success rate, while the best comparable model reaches 19.4%.
AI 根据报道生成 · 2 days ago更新
Reporting timeline
Follow the reporting to explore different perspectives.
- arXiv cs.ROFRED: World action model learns future video representations via self-distillation
The study proposes the FRED framework, which predicts future video representations from noise in a single step via self-distillation, using representations extracted from real future observations as targets, while historical observations provide temporal context for the video model. Across diverse real-world manipulation scenarios, FRED outperforms compared world action models, with notable gains on long-horizon and out-of-domain tasks under zero-shot environment generalization; using only human video demonstrations, it achieves a 50.0% success rate on tasks absent from robot training data, versus 19.4% for the best comparable model.