Learning the policy itself
REINFORCE, the Monte Carlo policy gradient: make each move more likely in proportion to how much better than usual its return was, with a baseline to cut noise.
Part of Policy gradients.
Read first
- Preferences, not values
Learn how much to favour each option, and compare every reward with what you usually get.
- Play it out, then average
Monte Carlo prediction estimates a state's value with no model by averaging the returns that follow each visit: unbiased, but it waits for the episode to end.
Read next
- A critic at every step
Actor-critic methods judge each move as it is made with a learned critic and its TD error, instead of waiting for the return: less luck, some bias.
- A centre and a spread
Gaussian policies for continuous actions: draw each action from a bell curve and let the policy gradient move its centre and learn how widely to explore.
- RLVR and GRPO: rewards you can check
How reinforcement learning with verifiable rewards (RLVR) trains reasoning models, and how GRPO scores each answer against its own group's mean.