Convexity
Why some losses have exactly one bottom and others have many — the chord, tangent and curvature tests, why a local minimum of a convex function is global, which ML losses are convex, and the two numbers (μ and L) that every convergence theorem is written in.
40 min
After this lesson you can
- State the definition of a convex set and a convex function, and test a function with the chord, tangent and Hessian tests
- Prove that the three tests are equivalent (for differentiable and twice-differentiable functions)
- Prove that every local minimum of a convex function is a global minimum
- Show which common ML losses are convex (least squares, logistic, hinge, lasso) and why neural-network losses are not
- Define μ-strong convexity and L-smoothness, and derive the bound
In this lesson
- Convex sets: every point can see every other
- Convex functions: the chord lies above the graph
- Three tests, one property
- Why a local minimum is a global one
- Which ML losses are convex?
- Strong convexity and smoothness: the two numbers
- In code
This lesson is part of Descent Pro
Descent Pro unlocks every lesson on the route, from base camp to the summit. Every plan starts with a free trial.