Gradient descent: step size and convergence
How gradient descent steps against the gradient, why the step size (learning rate) must stay under 2/L, and how the condition number sets its pace.
Part of Fitting a model.
Read first
- Convexity: why a convex loss has no bad minima
What makes a loss convex, why every local minimum of a convex function is global, and how to check with the second derivative (Hessian ⪰ 0).
Read next
- Logistic regression: sigmoid and log loss
How logistic regression turns a linear score into a probability with the sigmoid, and fits it by maximum likelihood: minimising log loss (cross-entropy).
- Momentum in SGD: why β = 0.9 speeds up training
How SGD with momentum keeps a running velocity of gradients, cancelling zigzags and mini-batch noise while speeding along steady directions.