Event
Egocentric Tennis Dataset Evaluates Vision Models for Spatiotemporal Prediction
1 reports · 1 sources
The research team collected and released an annotated egocentric tennis dataset, trained image encoders, video encoders, and vision-language models, and evaluated their performance on seven spatiotemporal prediction tasks. Results show that the best model localized the egocentric player in the current frame with an error of 0.14 meters and the ball with an error of 0.54 meters; trajectory prediction errors within 1.1 seconds were 0.34 meters and 0.91 meters, respectively, and the average error for hit-time prediction was 0.14 seconds. The team states that auxiliary depth information significantly improves 3D ball localization, indicating that inferring metric depth remains difficult; longer image history did not improve ball trajectory prediction, suggesting that the model relies more on scene context rather than only on inter-frame motion. The team also proposed a conditional diffusion model that generates full-body tennis kinematic motion from egocentric video at real-time rate. The dataset contains egocentric video and is synchronized with five third-person videos, estimated global positions, annotated hit times, and stroke types.
Report timeline
- Research team builds egocentric tennis dataset and evaluates vision models' spatiotemporal prediction capabilities
The research team collected an annotated egocentric tennis dataset and trained and evaluated image encoders, video encoders, and vision-language models on seven spatiotemporal prediction tasks. The best models achieved current-frame localization errors of 0.14 m for the egocentric player and 0.54 m for the ball, with trajectory prediction errors of 0.34 m and 0.91 m respectively within 1.1 seconds, and a mean error of 0.14 seconds for contact-time prediction. Auxiliary depth information notably improved 3D ball localization, while longer image histories did not improve ball trajectory prediction; the team also proposed a conditional diffusion model that generates full-body tennis kinematics from egocentric video at real-time rates.
arXiv cs.RO ·