Walk a few steps, then guess
Add up the next n rewards, then trust the value of where you land: a dial between TD(0) and Monte Carlo, where more steps mean less bias and more variance.
Part of Values.
Read first
- Play it out, then average
Monte Carlo prediction estimates a state's value with no model by averaging the returns that follow each visit: unbiased, but it waits for the episode to end.
- Temporal-difference learning
Nudge each guess toward the next one, and learn during the game, not after it.
Read next
- Learning the way by walking it
Sarsa, on-policy TD control: nudge each state–action value toward the reward plus the value of the move you take next, while choosing moves by those values.
Related
- Bias-variance tradeoff: why flexible models overfit
See why test error splits into bias², variance and noise, and why it is U-shaped as model complexity grows: the bias-variance tradeoff.