Logistic regression and softmax
When the target is a category, predict a probability instead — squash a linear score through a sigmoid, fit it by maximum likelihood (cross-entropy), and derive the beautifully simple gradient (p − y)·x. Then generalise to many classes with softmax.
40 min
After this lesson you can
- Explain why least squares on 0/1 labels makes a poor classifier
- Model with the sigmoid of a linear score, and interpret the weights as log-odds
- Derive binary cross-entropy from the Bernoulli likelihood, and its gradient
- Explain why there is no closed form but the problem is still convex, and what goes wrong on separable data
- Extend to classes with softmax and categorical cross-entropy, and derive the gradient with respect to the logits
In this lesson
- A line is a bad probability
- The sigmoid, and modelling a probability
- Fitting it: maximum likelihood gives cross-entropy
- The gradient: (p−y)x
- No closed form, but still a bowl
- Many classes: softmax regression
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.