Article

From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

The paper proposes a two-stage framework that learns complementary predictive representations from multi-view video clips to enhance embodied manipulation policies. In the pretraining stage, a frozen 3D foundation model provides dense geometric features, which are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution; action supervision is then applied to geometric transitions to extract a motor representation that emphasizes action-relevant transitions. In the downstream post-training stage, predictive geometry is transferred through geometry queries, and the motor representation guides intermediate features along the action-generation pathway. Future observations serve only as privileged supervision during training and are not needed at inference time.

Part of the event: Complementary Predictive Representation Framework Proposed to Improve Embodied Manipulation Policies

This page provides a summary and source link only.