PS2-RL: A Provably Safe Reinforcement Learning Framework Without Explicit Invariant Sets

Kai S. Yun, Zeyang Li, and Navid Azizan propose PS2-RL, a two-stage framework aimed at learning provably safe reinforcement learning policies without explicitly constructing control invariant sets. The authors state that the framework uses a learned backup policy to forward-integrate the system dynamics, generating implicit control invariant sets online to avoid the dimensionality expansion issues caused by explicit construction. In the first stage, a safe reachability value function is used to train the backup policy so that it can safely recover to a certified base set; in the second stage, the RL policy is trained end-to-end through a differentiable projection layer, which the authors claim strictly guarantees the safety induced by the backup policy. The authors state that the framework does not restrict the underlying RL algorithm, can be integrated into existing training pipelines, and has established theoretical guarantees. Evaluation was completed on robotic control tasks with state dimensions up to 10; the authors claim that prior provably safe RL methods are difficult to apply or become impractical in this dimensional range.

AI 根据报道生成 · 6 days ago更新