4
$\begingroup$

The derivation of the normal equation can be noted $\theta = (X^TX)^{-1}(X^T)y$, where $X^{-1}$ is the inverse of $X$ and can also be written $inv(X)$.

But why can't we write $inv(X^T X)$ as $inv(X)inv(X^T)$, and also use the definition of inverse $inv(X^T)(X^T) = I$?

Then $\theta = inv(X^T X)(X^T)y = inv(X)inv(X^T)(X^T)y = inv(X)Iy = inv(X)y$

Why isn't the final result of normal equation in regression in machine learning written as $\theta = inv(X) y$ after simplifying?

$\endgroup$
6
$\begingroup$

Only square matrices have an inverse, and neither $X$ nor $X^T$ here are (necessarily) square. I guess that's the most direct answer. $X$ may have a left-inverse $A$, such that $AX = I$. In that case, if $X\theta = y$ then $AX\theta = Ay$ so $\theta = Ay$.

The direct way to construct that $A$ is just where you started though; it's $(X^TX)^{-1}X^T$. The Gramian matrix $X^TX$ is (square, and) invertible if the features (columns) of $X$ are linearly independent, and, $X^TX$ is more importantly relatively quite small and so it's feasible to invert.

You normally wouldn't compute the inverse anyway, but try to solve the system $X^TX\theta = X^Ty$ with QR decomposition or something, but that's not what you're asking.

$\endgroup$
4
$\begingroup$

The value of $inv(X)$ or $inv(X^T)$ cannot be calculated for non-square matrices, and usually your number of samples and features is different, so your simplification is not generic.

However, you can simplify the functional notation in a very similar way to your suggestion using a pseudo-inverse - note that the link here is direct to the part of the article that describes this inverse as a direct solution to the linear least squares problem. Noting the pseudo inverse as $pinv(X)$ then:

$$\theta = pinv(X)y$$

However, this is simply hiding the more basic derivation behind a complex function. The computation required is essentially the same.

In practice if you are using matrix library that supports inverses and pseudo-inverses, you can choose to use a longer formulation using inverses or the pseudo-inverse version and it will give near identical results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.