PS2-RL: A Provably Safe and Scalable Reinforcement Learning Framework

The authors propose PS2-RL, a two-phase framework that uses a learned backup policy to forward-integrate the system dynamics and generate an implicit control-invariant set online, thereby avoiding the dimensionality-scaling problems caused by explicitly constructing control-invariant sets. In the first phase, the backup policy is trained with a safe-arrival value function to achieve safe recovery to a certified base set; in the second phase, an RL policy is trained end-to-end through a differentiable projection layer, strictly guaranteeing the safety induced by the backup policy. The framework does not restrict the underlying RL algorithm and can be integrated into existing training pipelines. It was evaluated on robot control tasks with state dimensions up to 10.