A critic at every step

Actor-critic methods judge each move as it is made with a learned critic and its TD error, instead of waiting for the return: less luck, some bias.

Part of Policy gradients.

Read first

  • Learning the policy itself

    REINFORCE, the Monte Carlo policy gradient: make each move more likely in proportion to how much better than usual its return was, with a baseline to cut noise.

Read next

  • A centre and a spread

    Gaussian policies for continuous actions: draw each action from a bell curve and let the policy gradient move its centre and learn how widely to explore.

  • Continuous actions

    DDPG (deep deterministic policy gradient) handles continuous actions: an actor climbs the critic's slope to the best action, with noise to explore nearby.

  • Stay close to the notes

    PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.