Event
VPM World Model Jointly Predicts Video and Point Trajectories
1 reports · 1 sources
The VPM research team proposes the Video-Point Model (VPM), which, conditioned on multi-view video history and language instructions, jointly predicts future video from arbitrary viewpoints and the 3D trajectories of persistent points in the scene. The team also built the VPM-Data simulation corpus (synchronized multi-view video paired with persistent point trajectories) and the VP-WorldBench prediction benchmark (covering both simulation and real environments). The team further incorporates robot actions into VPM as a token modality in the generator, so that the policy shares the point-supervised backbone; the paper states that in simulation and real-robot experiments, this action-enhanced model improves manipulation success rates.
Report timeline
- Video-Point Model learns 4D world through point trajectory prediction, improving robot manipulation success rate
The paper proposes Video-Point Model (VPM), a world model that conditions on multi-view video histories and language instructions to jointly predict future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. The authors add camera-conditioned multi-view processing and a trajectory prediction head to a pretrained world model, and build the VPM-Data simulation corpus and the VP-WorldBench benchmark; they further add robot actions as a token modality in the generator, so that the policy shares the point-supervised backbone.
HuggingFace Daily Papers ·