The closed form: normal equations

After this lesson you can

• Derive $X^{\top }X\hat{\mathbf{\theta }}=X^{\top }\mathbf{y}$ by setting the gradient of the squared error to zero
• Derive the same equations geometrically, as an orthogonal projection onto the column space of $X$
• Read every symbol of $\hat{\mathbf{\theta }}=(X^{\top }X{)}^{-1}{X}^{\top }\mathbf{y}$ and check its shapes
• Recognise when $X^{\top }X$ is singular or ill-conditioned, and what the pseudoinverse and minimum-norm solution do then
• Explain the $O(n{d}^2+d^3)$ cost and why numerical code uses QR or the SVD instead of an inverse

In this lesson

1. Setting up
2. Derivation 1: set the gradient to zero
3. Derivation 2: a right angle
4. Decoding (X⊤X)−1X⊤y
5. When the formula breaks
6. Cost, and what libraries actually do
7. In code