Long-horizon Q-learning effectively mitigates RL bootstrapping error

Scaling reinforcement learning to long horizons remains a major challenge. Long-horizon Q-learning (LQL) prevents the accumulation of bootstrapping error by constraining differences in the value function over long horizons. Compared with one-step TD and n-step returns, LQL shows significant improvement! Paper: https://arxiv.org/abs/2605.05812