Pretraining objectives: BERT, GPT and T5
How Transformers learn from raw text: next-word prediction (GPT), masked language modelling with the 80/10/10 rule (BERT), and span corruption (T5).
Part of Transformers.
Read next
- Scaling laws: compute-optimal LLMs (Chinchilla)
Why LLM loss falls as a power law in parameters and tokens, C ≈ 6ND, and the Chinchilla compute-optimal rule of about 20 tokens per parameter.
- Instruction tuning (SFT): teaching a model to answer
Instruction tuning, or supervised fine-tuning (SFT), trains a pretrained model on instruction–response pairs, scoring only the response tokens.