Temporal-difference learning

Nudge each guess toward the next one, and learn during the game, not after it.

Part of Values.

Read first

  • Explore or exploit

    Use what you know, or find out more: an agent that only does one does badly.

  • Reward versus value

    Reward is what’s good now; value is all the reward still to come. Choose by value.

Read next

  • Learning from every move

    Scoring each position teaches far faster than keeping the strategies that won.

  • Same notes, two answers

    Why batch Monte Carlo and TD(0) disagree on the same episodes: Monte Carlo fits the returns, TD(0) solves the model the data suggests (certainty equivalence).

  • Walk a few steps, then guess

    Add up the next n rewards, then trust the value of where you land: a dial between TD(0) and Monte Carlo, where more steps mean less bias and more variance.

  • Learning the way by walking it

    Sarsa, on-policy TD control: nudge each state–action value toward the reward plus the value of the move you take next, while choosing moves by those values.

Related