Learning from someone else’s trips
Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.
Part of Learning to act.
Read first
- Average, switch, and keep trying
Monte Carlo control learns to act from whole episodes: average each move's return, switch to the best, and keep exploring with exploring starts or ε-greedy.
Read next
- Same notes, two answers
Why batch Monte Carlo and TD(0) disagree on the same episodes: Monte Carlo fits the returns, TD(0) solves the model the data suggests (certainty equivalence).
- Stay close to the notes
PPO (proximal policy optimization) reuses one batch for many updates by clipping the probability ratio at 1 ± ε, keeping the policy close to the old one.
Related
- Learning the best while still exploring
Q-learning backs up the best next move instead of the one taken, so it learns optimal values while still exploring (off-policy TD control).