University of Auckland and Xi'an Jiaotong University release Video2Knowledge framework to extract robot task knowledge from monocular phone video

The LISMS lab at the University of Auckland and Xi'an Jiaotong University jointly proposed the Video2Knowledge framework, which extracts 6D hand and object trajectories from an ordinary monocular RGB video and converts them into robot action sequences. The system uses WiLoR to recover 3D hand meshes, combines OnePoseViaGen for object tracking, and designs a four-stage physics-aware correction module; manipulated-object pose accuracy rises from 52.5% to 59.0% under frontal views.