Perplexity: how language models are scored
Perplexity scores a language model by the per-word chance it gives held-out text, read as how many words it is choosing between at each step.
Part of Predicting the next word.
Read next
- Recurrent neural networks (RNNs) and vanishing gradients
How a recurrent neural network (RNN) language model reads any length with shared weights, trains by backpropagation through time, and why gradients vanish.
- Scaling laws: compute-optimal LLMs (Chinchilla)
Why LLM loss falls as a power law in parameters and tokens, C ≈ 6ND, and the Chinchilla compute-optimal rule of about 20 tokens per parameter.