Event

Study: bf16 fine-tuning of VLA silently drops updates

1 reports · 1 sources

A paper by Shubham Singh points out that if vision-language-action (VLA) models are fine-tuned with bf16 full weights and no fp32 master copy is kept, AdamW will directly round away updates smaller than 1/512 of the weight. The paper says that 9 of 12 open-source VLA codebases train this way by default or in documented recipes, including LeRobot's SmolVLA default configuration and the π0/π0.5 guides. In LIBERO experiments, when there was no fp32 copy, probing every 100 steps found that at least 79% of bf16 weight updates were erased. On LIBERO-Object, keeping an fp32 copy raised SmolVLA's success rate from 4.3% to 49.3%, and π0.5 at a ten-times lower learning rate from 5.1% to 80.9% (both were undertrained); π0.5 at the default learning rate rose from 89.0% to 97.7%. For SmolVLA at the default learning rate, open-loop 50-action chunks improved by 10.2 percentage points, while replanning every 10 steps improved by only 4.2 percentage points. The fix proposed in the paper is to keep trainable weights in fp32 during training; for SmolVLA in LeRobot this is a one-line change.