Random forests: bagging, bootstrap and decorrelated trees
Why averaging trees helps: bagging (bootstrap aggregating), out-of-bag error, and how random forests lower tree correlation ρ in ρσ² + (1−ρ)σ²/B.
Part of Trees and ensembles.
Read first
- Bias-variance tradeoff: why flexible models overfit
See why test error splits into bias², variance and noise, and why it is U-shaped as model complexity grows: the bias-variance tradeoff.
- Decision trees: Gini, entropy and greedy splits
How a decision tree picks each split greedily by Gini impurity or entropy, and why a tree grown until every leaf is pure overfits.
Read next
- k-means clustering and k-means++
How k-means clustering groups unlabelled points: Lloyd's algorithm never raises its objective but can stick at a local optimum; k-means++ starts it well.