Difference between Sum of Squares and Maximum Likelihood Linear Regression

Question

I'm new in Machine Learning and one of the first arguments I'm studying is linear regression. I understood that , in few words , the idea to use the Linear Regression is to learn an hypothesis that can map a new input x in a good approximation of y.

In order to do this , if my hypothesis is :

h(x) = wx + w0

I have to update my parameters minimizing an error function like Least Squares and optimize the w vector with the help of an optimization algorithm like Gradient Descent.

I understood how this works , but sometimes I see this "Maximum Likelihood Estimation" and I did not understand if it is another way to estimate the w parameters or something else.

Answer 1

Suppose you construct a probabilistic model, where $y_i$ are believed to be related to your $x_i$ under the formula;

$$ y_i = w_1 x_i + w_0 + \epsilon_i $$

So $w_1$ and $w_0$ are your target parameters from before and $\epsilon_i$ is an error term which you expect follows some probability distibution, e.g. $\mathcal{N}(0,\sigma^2)$. It is important that its expectation is zero.

Given your data you want to maximise the probability of returning every $y_i$ given your data $x_i$ subject to your model. The probability of getting $y_i$ from $x_i$ is equal to $f_{\epsilon}(y_i-w_1x_i-w_0)$, where $f_{\epsilon}$ is the probability density function of $\epsilon$, i.e. a normal distribution.

And the likelihood of getting every $y_i$ is the product of them all,

$$ L = \prod_i f_{\epsilon}(y_i-w_1x_i-w_0) $$

You want to maximise this value by tweaking $w_1$ and $w_0$, hence the name maximimum likelihood estimation.

Note that this is equivalent to maximising the log likelihood, so;

$$ \log L = \sum_i \log f_{\epsilon}(y_i-w_1x_i-w_0)$$

And if you look at the normal distribution density function you will see that (after ignoring some constants) this reduces to the problem of maximising..

$$ - \sum_i (y_i-w_1x_i-w_0)^2 $$

or in other words minimising the sum of squares akin to OLS.

But like using a different distance function in OLS you can parametrise a different error distribution in MLE.

Answer 2

Just to add to the previous response, maximum likelihood estimation is in fact a very general procedure for estimating parameters of an underlying distribution $f_\theta$ using data $\{ x_i \}_{i=1}^{n}$ assumed to arise from this distribution, where $\theta$ is an unknown set of constants that parameterizes the distribution $f$. Specifically, maximum likelihood estimation seeks to find values for the parameters that render our data the most probable ex post facto. The technique is popular both because of it's intuitive appeal, and because maximum likelihood estimates enjoy certain large sample optimality properties.

As @Attack68 pointed out, the least squares estimates for the coefficients in linear regression turn out to be maximum likelihood estimates in the unique case where the error distribution is Gaussian. If instead we were to use another distribution such as the Laplacian then our likelihood function becomes

$$ \prod_{i=1}^{n} \frac{1}{2 \sigma} \exp \left ( - \frac{|y_i - \beta^T x_i|}{\sigma} \right ) = \frac{1}{(2 \sigma )^n} \exp \left ( - \frac{ \sum_{i=1}^{n} |y_i - \beta^T x_i|}{\sigma} \right ) . $$

Now we see that instead of minimizing the sum of squares, the maximum likelihood estimates for $\beta$ are based on the absolute residuals. It all depends on what kind of parametric assumptions we make about the underlying data generating process.