Event

Proposing a Language Feedback Policy that Provides Harmless Guidance for Frozen VLAs

1 reports · 1 sources

Hyun Joe Jeong, Gokul Swamy, and Andrea Bajcsy propose a framework that uses interactive search to find language sequences that improve closed-loop VLA task performance, distills them into a test-time language feedback policy (LFP), and trains an improvement head to predict when language guidance will be effective; the improvement head is then conformally calibrated to prevent harmful interventions. The method applies to any frozen pretrained VLA, requires no access to the original training distribution, and does not fine-tune the underlying model. The paper reports that in seen environments, the conformally calibrated LFP improves base VLA performance by 24.7% (simulation) and 65.0% (hardware); under visual and semantic perturbations for which calibration samples are available, it provides strong harmlessness guarantees and produces recovery behaviors not seen in open-loop prompting. The authors also note that in out-of-distribution scenarios, LFP reduces task performance relative to the original instructions, which is precisely why conformal calibration is introduced.