Transformer block: residuals, layer norm, encoder–decoder
How a Transformer block wraps attention and a feed-forward network in residual connections and layer norm, and how stacks of blocks become BERT, GPT and T5.
Part of Transformers.
Read next
- Subword tokenization: byte-pair encoding (BPE)
How byte-pair encoding (BPE) builds a subword vocabulary by merging the most frequent pair, so a model can split any new word into known pieces.
- Vision Transformer (ViT): images as 16×16 patches
How the Vision Transformer (ViT) cuts an image into patches, turns each into a token with a [class] token, and why it needs far more data than a CNN.
Related
- Batch normalization: why networks normalize each batch
How batch normalization (BatchNorm) evens out each channel over a mini-batch, what the learned γ and β do, and why it uses running averages at test time.