Learning from someone else’s trips

Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.

Part of Learning to act.

Read first

  • Average, switch, and keep trying

    Monte Carlo control learns to act from whole episodes: average each move's return, switch to the best, and keep exploring with exploring starts or ε-greedy.

Read next

  • Same notes, two answers

    Why batch Monte Carlo and TD(0) disagree on the same episodes: Monte Carlo fits the returns, TD(0) solves the model the data suggests (certainty equivalence).

  • Stay close to the notes

    PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.

Related