OPO: Policy optimization via adaptive clipping ranked by importance ratios
What happened
Dongkyu Derek Cho and Pan Xu propose Ordered Policy Optimization (OPO). Addressing the problem that PPO uses a fixed clipping range for all samples and cannot distinguish extreme importance ratios from tolerable large changes, OPO instead sorts importance ratios within each minibatch, selects the extreme ratios that most affect the update, and assigns them adaptive per-sample clipping radii according to a preset tail profile, rather than a uniform fixed radius. The authors state that the objective remains first-order and requires only minibatch sorting and a modified policy loss; experiments on MuJoCo and Atari benchmarks show that this change achieves strong performance relative to existing baselines.
AI 根据报道生成 · 2 days ago更新
Reporting timeline
Follow the reporting to explore different perspectives.
- arXiv cs.ROOrdered Policy Optimization (OPO): A reinforcement learning policy optimization method with adaptive clipping ordered by importance ratios
Addressing the problem that PPO uses a fixed clipping interval for all samples and cannot distinguish extreme importance ratios from tolerable large changes, the authors propose Ordered Policy Optimization (OPO): within each minibatch, it sorts the importance ratios, selects the extreme ratios that have the greatest effect on the update, and assigns them adaptive per-sample clipping radii according to a preset tail profile. The objective remains first-order and requires only minibatch sorting and a modified policy loss.
Activity for this event
- Current
- 3
- Peak
- 10Nov 9 14:00
- Past 24 hours
- -50%
The trend compares participants observed throughout the same period, so its range may be smaller than the current interest count. Move or tap to inspect each hour; use the arrow keys on a keyboard.