Stay close to the notes

PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.

Part of Policy gradients.

Read first

  • A critic at every step

    Actor-critic methods judge each move as it is made with a learned critic and its TD error, instead of waiting for the return: less luck, some bias.

  • Learning from someone else’s trips

    Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.

Read next

  • Continuous actions

    DDPG (deep deterministic policy gradient) handles continuous actions: an actor climbs the critic's slope to the best action, with noise to explore nearby.

Related