Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
The derivation of the normal equation can be noted $\theta = (X^TX)^{-1}(X^T)y$, where $X^{-1}$ is the inverse of $X$ and can also be written $inv(X)$.
But why can't we write $inv(X^T X)$ as $inv(X)inv(X^T)$, and also use the definition of inverse $inv(X^T)(X^T) = I$?
Then $\theta = inv(X^T X)(X^T)y = inv(X)inv(X^T)(X^T)y = inv(X)Iy = inv(X)y$
Why isn't the final result of normal equation in regression in machine learning written as $\theta = inv(X) y$ after simplifying?
Only square matrices have an inverse, and neither $X$ nor $X^T$ here are (necessarily) square. I guess that's the most direct answer. $X$ may have a left-inverse $A$, such that $AX = I$. In that case, if $X\theta = y$ then $AX\theta = Ay$ so $\theta = Ay$.
The direct way to construct that $A$ is just where you started though; it's $(X^TX)^{-1}X^T$. The Gramian matrix $X^TX$ is (square, and) invertible if the features (columns) of $X$ are linearly independent, and, $X^TX$ is more importantly relatively quite small and so it's feasible to invert.
You normally wouldn't compute the inverse anyway, but try to solve the system $X^TX\theta = X^Ty$ with QR decomposition or something, but that's not what you're asking.
The value of $inv(X)$ or $inv(X^T)$ cannot be calculated for non-square matrices, and usually your number of samples and features is different, so your simplification is not generic.
However, you can simplify the functional notation in a very similar way to your suggestion using a pseudo-inverse - note that the link here is direct to the part of the article that describes this inverse as a direct solution to the linear least squares problem. Noting the pseudo inverse as $pinv(X)$ then:
$$\theta = pinv(X)y$$
However, this is simply hiding the more basic derivation behind a complex function. The computation required is essentially the same.
In practice if you are using matrix library that supports inverses and pseudo-inverses, you can choose to use a longer formulation using inverses or the pseudo-inverse version and it will give near identical results.
