Building photorealistic simulation environments with augmentable Gaussian splats to train robot policies

The author turned to high-quality Gaussian Splats of real sites to build augmentable, relightable photorealistic simulation environments, because policies trained in animated simulation were hard to transfer to real machines. The pipeline, refined over hundreds of iterations, converts iPhone scans into Splats plus meshes, selects robot-operable assets, and splits them by unit to generate detailed assets placed back at their original positions in the Splat. Physics and rendering use MuJoCo Warp, and the robot is a Franka Panda.