Event
FELT uses RGB to generate tactile signals, improving real-robot task success rates by 15-22.5 points
1 reports · 1 sources
Zinan Li et al. propose the FELT framework, which leverages existing visuo-tactile paired data to learn contact-aware representations from RGB observations and provides them to downstream policies through two interfaces: generated tactile images and contact-aware features. On four contact-intensive real-robot tasks, both interfaces achieve higher success rates than vision-only policies, and directly integrating contact-aware features into the policy improves success rates by 15-22.5 percentage points. FELT employs a frozen visual encoder, together with separate query decoders for the left and right fingers; the two branches exchange information via gated connections, and a convolutional readout generates spatial contact representations corresponding to the spatial arrangement of the dual-finger tactile arrays.
Report timeline
- FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation
FELT is a framework that uses existing paired visuo-tactile data to learn contact-aware representations from RGB observations, providing them to downstream policies through two interfaces: generated tactile images and contact-aware features. It combines a frozen visual encoder with separate query decoders for the left and right fingers, and the two branches exchange information through gated connections, with a convolutional readout producing spatial contact representations. On four contact-intensive real-robot tasks, both interfaces achieved higher success rates than vision-only policies, and directly integrating contact-aware features into policies raised success rates by 15-22.5 percentage points.
arXiv cs.RO ·