Note Set 15b: The Amazing Normal Equation

We use the idea of projection to determine the best approximate solution to $\mathbf{A}\vec{x}=\vec{b}$. The beautiful story is that we find the part of $\vec{b}$ that lives in Column Space, $\vec{p}$, and trim off the Left Nullspace part, $\vec{e}$. We find out that every $\mathbf{A}\vec{x}=\vec{b}$ problem can be turned into a solvable, square symmetric system: $\mathbf{A}^{\mbox{T}}\mathbf{A}\vec{x}=\mathbf{A}^{\mbox{T}}\vec{b}$.
Best consumed by 2016/10/12