Subword tokenization: byte-pair encoding (BPE)
How byte-pair encoding (BPE) builds a subword vocabulary by merging the most frequent pair, so a model can split any new word into known pieces.
Part of Transformers.
Read next
- Pretraining objectives: BERT, GPT and T5
How Transformers learn from raw text: next-word prediction (GPT), masked language modelling with the 80/10/10 rule (BERT), and span corruption (T5).