Event

FERPO: An Entropy-Regularized Policy Optimization Algorithm Without Relying on Critic Action Gradients

1 reports · 1 sources

Sebastian Sanokowski and others proposed FERPO (Forward Entropy-Regularized Policy Optimization), an online maximum-entropy reinforcement learning algorithm that improves the policy by differentiating the critic's value rather than its actions, avoiding policy update problems caused when value predictions are accurate but action derivatives are unreliable; the actor is then fitted with a forward KL objective, estimated via self-normalized importance sampling (SNIS). Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample efficiency improvements; computational benchmarks show that its actor updates are faster than REPPO (Relative Entropy Pathwise Policy Optimization).