Batch normalization: why networks normalize each batch
How batch normalization (BatchNorm) evens out each channel over a mini-batch, what the learned γ and β do, and why it uses running averages at test time.
Part of Learning to see.
Read next
- Semantic segmentation: a label for every pixel
How semantic segmentation labels every pixel: fully convolutional networks (FCN), transposed convolutions, checkerboards, and dilated convolutions.
Related
- Transformer block: residuals, layer norm, encoder–decoder
How a Transformer block wraps attention and a feed-forward network in residual connections and layer norm, and how stacks of blocks become BERT, GPT and T5.