Event
GenSight: Unified Video Generation and 4D Perception World Model
1 reports · 1 sources
Haitang Feng and other authors released a paper proposing GenSight, a unified embodied world model that integrates future video generation and dense perception into a single pretrained video diffusion Transformer. The model uses a unified spatiotemporal representation covering video, depth, embodied segmentation, and 2D optical flow relative to the first frame; each modality is predicted on a shared DiT backbone via lightweight task modulation and jointly optimized in a unified velocity prediction space. At inference time, two modes are supported: multi-step future video generation plus aligned dense prediction, and single-step dense perception from an observed video. The training data covers embodied manipulation data from a variety of robot embodiments and human hand interaction.
Report timeline
- GenSight: A Unified Embodied World Model for Video Generation and 4D Perception
GenSight is a unified embodied world model that integrates future video generation and dense perception into a single pretrained video diffusion Transformer. It uses a unified spatiotemporal representation covering video, depth, embodiment segmentation, and 2D optical flow relative to the first frame; built on a shared DiT backbone, it predicts each modality within the same architecture via lightweight task modulation, and jointly optimizes them in a unified velocity prediction space. At inference, it supports multi-step future video generation with aligned dense prediction, as well as single-step dense perception from an observed video. Training data covers a variety of robot embodiments and human hand interaction.
arXiv cs.RO ·
Related topics
Interest trend
- Current
- 3
- Peak
- 4Nov 4 01:00
- Past 24 hours
- –
The trend compares participants observed throughout the same period, so its range may be smaller than the current interest count. Move or tap to inspect each hour; use the arrow keys on a keyboard.