Top-k, top-p (nucleus) sampling and temperature
How language models pick the next word: sampling, top-k, top-p (nucleus) sampling and temperature, and why cutting the tail keeps text varied but sensible.
Part of Translation and generation.
Read next
- BLEU score: how machine translation is evaluated
How BLEU (Bilingual Evaluation Understudy) scores generated text: clipped n-gram precision, the geometric mean and the brevity penalty, and where it fails.
Related
- Chain-of-thought prompting and self-consistency
Why letting an LLM write its steps first (chain-of-thought prompting) solves more word problems, and how voting over sampled chains (self-consistency) helps.