Attention in seq2seq: soft alignment for translation
How the attention mechanism lets an encoder–decoder build a fresh context vector at each step: scores, softmax weights, and attention as soft alignment.
Part of Translation and generation.
Read next
- Beam search: decoding beyond greedy
Why greedy decoding can miss the likeliest sentence, how beam search keeps the k best hypotheses, length normalisation, and why it repeats itself in open text.
- Self-attention: queries, keys, values and masking
Self-attention lets every word attend to every word through queries, keys and values; why it needs position vectors, and how a decoder masks the future.