Event
Study: Human Behavior Model Evaluation Metrics Struggle to Predict Robot Policy Utility
1 reports · 1 sources
Haoyang Jiang and other authors published a study that, in two scenarios—high-speed driving and object handover—used common generative model metrics to evaluate multiple types of human behavior models, then compared these with the downstream performance of reinforcement learning policies trained using these models. The results show that existing behavior evaluation metrics cannot reliably predict downstream policy performance, and that evaluation results are sensitive to the trajectory representation method. The study also states that even when the true test behavior distribution is known, training with a different distribution may yield better downstream performance.
Report timeline
- Study: Behavioral quality of human behavior models does not equal robot learning utility
The paper evaluates multiple types of human behavior models using common generative model metrics in two scenarios, high-speed driving and object handover, and compares them with the downstream performance of reinforcement learning policies trained using these models. The results show that existing behavioral evaluation metrics cannot reliably predict downstream policy performance, and that evaluation results are sensitive to the trajectory representation; even when the true test behavior distribution is known, training with a different distribution may yield better downstream performance.
arXiv cs.RO ·