Self-attention: queries, keys, values and masking
Self-attention lets every word attend to every word through queries, keys and values; why it needs position vectors, and how a decoder masks the future.
Part of Transformers.
Read first
- Attention in seq2seq: soft alignment for translation
How the attention mechanism lets an encoder–decoder build a fresh context vector at each step: scores, softmax weights, and attention as soft alignment.
Read next
- Multi-head attention and the √d_k scale
Why Transformers divide attention scores by √d_k (dot products grow and saturate the softmax) and split attention into multi-head attention.