Difference between coefficient of determination and least squared error?

Question

Where and when should I consider R^2 as a measure of goodness-of-fit for regressions?
Usually I choose the least squared model as the best model.
Is it possible that the least squared model does not have the highest R squared?
Should I use both as the evaluation of regressions?
Could you give me an intuitive example?

Answer 1

R squared is one of many model diagnostics. One calculation of it is (1-var(residuals)/var(response variable)). Since the response variable is constant, you want to minimize the average squared residuals (the variance), which is identical to least squares.

The issue with using only R-squared is that if you have a lot of variables in a model, you will likely get a high R-squared, but that doesn't mean the model is good. In fact, if you have 100 rows of data, 100 independent variables and 1 response variable, there's a good chance you will get an R-squared of 1, but the model will be completely overfit.

There are 2 common solutions to this problem:

1. Add a penalty term that reduces (or increases, depending on what is optimal) the "score" of a model when it has more variables.
2. Use a form of validation to see if the model performs well on data it wasn't trained on.

Oftentimes both of these are used to varying degrees, but here are a few examples:

1. AIC, or Aikaike Information Criterion, is -2*log(P(Y|X)) + 2*p, where P(Y|X) is the probability density of Y given X under the current model and p is the number of predictor (X) variables. The goal is to minimize AIC.

2. In K-fold cross-validation, where K is the number of subsets you divide data into, you train the data on K-1 of the sets combined and then test it on the remaining set and report the error. You repeat this process for each combination of K-1 training sets and 1 remaining set and combine the K results (usually by averaging error).

The first method is usually a safe bet for linear regression models, but oftentimes, especially when you are using more complex algorithms, you will want to validate the algorithm to make sure it isn't overfitting the training data. Also, using validation allows comparisons between different models (especially non-parametric ones) to be made much more easily.

For more ways of evaluating models, I would suggest reading into different regularization techniques and different validation techniques.