Maximum likelihood: where losses come from
Choose the parameters under which the data you actually saw is most probable. That one principle gives you squared error, cross-entropy and, once you add a prior, weight decay.
50 min
After this lesson you can
- Distinguish probability from likelihood, and explain why we maximise the log-likelihood
- Derive the maximum-likelihood estimates for a coin and for a Gaussian
- Show that minimising squared error is maximum likelihood under Gaussian noise
- Show that cross-entropy is the negative log-likelihood of Bernoulli and categorical outputs
- Derive MAP estimation and show that a Gaussian prior is L2 regularisation
In this lesson
- Probability and likelihood: one formula, two readings
- A coin, by hand
- A Gaussian, by hand
- Squared error is maximum likelihood with Gaussian noise
- Cross-entropy is maximum likelihood for classes
- Priors: MAP estimation and regularisation
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.