Constraints, projections and proximal methods
One idea for walls and corners — take a gradient step, then fix it up. Projected gradient descent, the proximal operator, soft-thresholding and why the lasso lands on exact zeros, ISTA and FISTA, mirror descent, and weight decay in AdamW read as a proximal step.
40 min
After this lesson you can
- Implement projected gradient descent and derive projections onto a box, a ball and the probability simplex (the last via KKT)
- Define the proximal operator and derive the proximal gradient step as the minimiser of a linearised model plus the non-smooth term
- Derive soft-thresholding as the proximal operator of the ℓ1 norm and explain why proximal methods produce exact zeros while subgradient descent does not
- State the convergence rates of ISTA and FISTA and implement both
- Explain mirror descent with a Bregman divergence, and the exponentiated-gradient update on the simplex
- Read decoupled weight decay (AdamW) as a proximal step, and say why it differs from an L2 penalty under Adam
In this lesson
- Projected gradient descent
- The proximal operator
- Soft-thresholding: why the lasso gives exact zeros
- ISTA and FISTA
- Mirror descent: change the geometry, not just the step
- Weight decay is a proximal step
- Where else you will meet this
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.