Stay close to the notes
PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.
Part of Policy gradients.
Read first
- A critic at every step
Actor-critic methods judge each move as it is made with a learned critic and its TD error, instead of waiting for the return: less luck, some bias.
- Learning from someone else’s trips
Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.
Read next
- Continuous actions
DDPG (deep deterministic policy gradient) handles continuous actions: an actor climbs the critic's slope to the best action, with noise to explore nearby.
Related
- DPO vs RLHF: learning from human preferences
How Direct Preference Optimization (DPO) trains a model on human preference pairs with one loss, reaching RLHF's result with no reward model or RL loop.
- Proximal Policy Optimization Algorithms
Schulman et al., 2017.