0
$\begingroup$

When trying to fit a line to data, using linear regression, we would like it to have the lowest sum of squared differences (least squares method).

I am wondering why we use this technique instead of finding the lowest line which has the lowest variance.

Is there a difference between averaging the sum of squared difference and not averaging it? Why would we not just get rid of the least squares method and just find variance to find line of best fit?

$\endgroup$
0
$\begingroup$

I did not understand your question but I will try to answer to the two possible meanings of your answer.

Case 1:

I am wondering why we use this technique instead of finding the lowest line which has the lowest variance

Here, I understand that you want to ask why is not a "minimum variance" method used. The equation of the "line" with minimum variance is always a straight line parallel to the $x$ axis -the mean- (variance = 0).

Case 2:

Is there a difference between averaging the sum of squared difference and not averaging it?

Here, I understand that you want to get rid of the averaging term of the equation. You can get rid of it without any problem, it's a constant and does not have any influence in the result.

Why would we not just get rid of the least squares method and just find variance to find line of best fit?

The reason of why we don't get rid of it is because is the only differentiable method to minimize the error between the regression and the data, other methods exist but are skewed estimators

$\endgroup$
0
$\begingroup$

Which variance are you referring to? MSE is composed of variance and bias (squared). Here, variance refers to variance of the estimator, and OLS while minimizing the MSE also minimizes the variance of regression coefficients. That's why it's called BLUE (Best Linear Unbiased Estimator), with "Best" referring to lowest possible variance.

If you are referring to minimizing the variance of residuals, then minimizing the sum of squared residuals is better because "adding a constant to all the predictions doesn't change the variance of the residuals, so this objective function never tells you what the intercept should be." (https://qr.ae/TWIIjC)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.