Learning the best while still exploring

Q-learning backs up the best next move instead of the one taken, so it learns optimal values while still exploring (off-policy TD control).

Part of Learning to act.

Read first

  • Learning the way by walking it

    Sarsa, on-policy TD control: nudge each state–action value toward the reward plus the value of the move you take next, while choosing moves by those values.

Read next

  • The lucky favourite

    Maximization bias: the best of several noisy estimates is overrated, so Q-learning's max overestimates, and Double Q-learning splits picking from scoring.

  • Where your move leaves you

    Afterstates score the situation your move leaves you in, not every state–action pair, so every way of reaching it shares one value and learns faster.

  • Deep Q-networks

    Deep Q-networks (DQN) replace Q-learning's table with a neural network, steadied by experience replay and a frozen target network.

Related

  • Learning from someone else’s trips

    Off-policy evaluation with importance sampling: weight each return by how much likelier your policy makes its moves. Ordinary is unbiased; weighted is steadier.