Temporal-difference learning
Nudge each guess toward the next one, and learn during the game, not after it.
Part of Values.
Read first
- Explore or exploit
Use what you know, or find out more: an agent that only does one does badly.
- Reward versus value
Reward is what’s good now; value is all the reward still to come. Choose by value.
Read next
- Learning from every move
Scoring each position teaches far faster than keeping the strategies that won.
- Same notes, two answers
Why batch Monte Carlo and TD(0) disagree on the same episodes: Monte Carlo fits the returns, TD(0) solves the model the data suggests (certainty equivalence).
- Walk a few steps, then guess
Add up the next n rewards, then trust the value of where you land: a dial between TD(0) and Monte Carlo, where more steps mean less bias and more variance.
- Learning the way by walking it
Sarsa, on-policy TD control: nudge each state–action value toward the reward plus the value of the move you take next, while choosing moves by those values.
Related
- Temporal Difference Learning and TD-Gammon
Gerald Tesauro, 1995.
- Where your move leaves you
Afterstates score the situation your move leaves you in, not every state–action pair, so every way of reaching it shares one value and learns faster.