Ordered Policy Optimization (OPO): A reinforcement learning policy optimization method with adaptive clipping ordered by importance ratios

Addressing the problem that PPO uses a fixed clipping interval for all samples and cannot distinguish extreme importance ratios from tolerable large changes, the authors propose Ordered Policy Optimization (OPO): within each minibatch, it sorts the importance ratios, selects the extreme ratios that have the greatest effect on the update, and assigns them adaptive per-sample clipping radii according to a preset tail profile. The objective remains first-order and requires only minibatch sorting and a modified policy loss.