Sooner is worth more
Count each reward further off for less, and even a task that never ends has a finite return.
Part of Markov decision processes.
Read first
- Reward versus value
Reward is what’s good now; value is all the reward still to come. Choose by value.
Read next
- A square is worth its neighbours
A state is worth the average of reward plus the discounted worth of where you land: the Bellman equation.
- Play it out, then average
Monte Carlo prediction estimates a state's value with no model by averaging the returns that follow each visit: unbiased, but it waits for the episode to end.