Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

The paper identifies two types of decode amplification when randomly sampling videos for AI training: sequential reconstruction of unrelated frames, and reconstruction of the reference chains needed for sampled targets. The authors propose dependency-aware sparse decoding, which reconstructs only the requested frames and their necessary references without transcoding the source video; optional offline DDRA encoding constrains each interior frame between two intra-coded anchors, so that reconstruction of each requested target requires no more than three frames. On a large robotics corpus, the method achieves 1.45x the loading throughput of LeRobot v3 at similar reconstruction quality, with 58.7% less media data.