A critic at every step
Actor-critic methods judge each move as it is made with a learned critic and its TD error, instead of waiting for the return: less luck, some bias.
Part of Policy gradients.
Read first
- Learning the policy itself
REINFORCE, the Monte Carlo policy gradient: make each move more likely in proportion to how much better than usual its return was, with a baseline to cut noise.
Read next
- A centre and a spread
Gaussian policies for continuous actions: draw each action from a bell curve and let the policy gradient move its centre and learn how widely to explore.
- Continuous actions
DDPG (deep deterministic policy gradient) handles continuous actions: an actor climbs the critic's slope to the best action, with noise to explore nearby.
- Stay close to the notes
PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.