Learning the best while still exploring
Q-learning backs up the best next move instead of the one taken, so it learns optimal values while still exploring (off-policy TD control).
Part of Learning to act.
Read first
- Learning the way by walking it
Sarsa, on-policy TD control: nudge each state–action value toward the reward plus the value of the move you take next, while choosing moves by those values.
Read next
- The lucky favourite
Maximization bias: the best of several noisy estimates is overrated, so Q-learning's max overestimates, and Double Q-learning splits picking from scoring.
- Where your move leaves you
Afterstates score the situation your move leaves you in, not every state–action pair, so every way of reaching it shares one value and learns faster.
- Deep Q-networks
Deep Q-networks (DQN) replace Q-learning's table with a neural network, steadied by experience replay and a frozen target network.
Related
- Learning from someone else’s trips
Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.