Subword tokenization: byte-pair encoding (BPE)

How byte-pair encoding (BPE) builds a subword vocabulary by merging the most frequent pair, so a model can split any new word into known pieces.

Part of Transformers.

Read next