Gaussian mixtures and EM
Give every cluster its own shape and let points belong partly to several. The result is a probability model, the Gaussian mixture, and fitting it leads to the expectation–maximisation algorithm, whose every step provably raises the likelihood.
55 min
After this lesson you can
- Write the Gaussian mixture density and explain it as a two-step generative story
- Compute responsibilities with Bayes' rule and interpret them as soft assignments
- Derive the EM updates for a Gaussian mixture
- Prove, with Jensen's inequality, that EM never decreases the log-likelihood
- Recover k-means as a limiting case, explain the likelihood singularity, and choose K with BIC
In this lesson
- A story that generates the data
- Fitting: a chicken-and-egg problem
- Why EM can only climb
- In code
- k-means is EM with the dials turned down
- Singularities: when the likelihood lies
- How many components? BIC
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.