Evaluation metrics for Decision Tree regressor and KNN regressor

Question

I have started working on the Decision Tree Regressor and KNN Regressor.

I have built the model and not sure what are the metrics needs to be considered for evaluation. As of now I have considered Root mean squared error.

Can we use $R^2$ values for Decision Tree and KNN or is it applicable only to Linear Regression Model

I have also pasted the $R^2$ and MSE values. As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?

R2 Score for train - KNN regression 0.8215942683102192
R2 Score for test - KNN regression 0.7160388084850589
Mean squared error for train - KNN regression 49.92162362176166
Mean squared error for test - KNN regression 78.30907381395349

R2 Score for train - linear regression 0.6141419744748021
R2 Score for test - linear regression 0.6117893766210736
Mean squared error for train - linear regression 107.97107771851036
Mean squared error for test - linear regression 107.0583420197463

R2 Score for train - Decision Tree regression 0.9962039204515297
R2 Score for test - Decision Tree regression 0.7866182225490949
Mean squared error for train - Decision tree regression 1.0622217832469776
Mean squared error for test  - Decision tree regression 58.84511637596899

Updating the AIC and BIC values

AIC value - Test - KNN       :   -3.8272461328797505
BIC value - Test - KNN       :   1169.4748549110836
AIC value - Test - Linear Reg:   -4.452667046616746
BIC value - Test - Linear Reg:   1250.154152783156
AIC value - Test - Decision T:   -3.2766787253336602
BIC value - Test - Decision T:   1098.4516593376377

Thank you.

Answer 1

Generally when ever we are trying to compare between models and to choose the best one, we go for other metrics like AIC, BIC, AUC(this is not applicable as it is used for classification algorithm) etc along with $R^2$.

Now why are they important criteria because AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered. On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.

Generally we use both AIC and BIC together.

you can got through this Link to understand better on AIC, BIC and which is better but conclusion is both are important.

Now when we compare between $R^2$ and AIC value:

$R^2$ and AIC are answering two different questions. I want to keep this breezy and non-mathematical, so my statements are non-mathematical. $R^2$ is saying something to the effect of how well your model explains the observed data. If the model is regression and non-adjusted $R^2$ is used, then this is correct on the nose.
AIC, on the other hand, is trying to explain how well the model will predict on new data. That is, AIC is a measure of how well the model will fit new data, not the existing data. Lower AIC means that a model should have improved prediction.

Frequently, adding more variables decreases predictive accuracy and in that case the model with higher $R^2$ will have a higher (worse) AIC. A nice example of this is in "Introduction to Statistical Learning with R" in the chapter on regression models including 'best subset' and regularization. They do a pretty thorough analysis of the 'hitters' data set. One can also do a thought experiment. Imagine one is trying to predict output on the basis of some known variables. Adding noise variables to the fit will increase $R^2$, but it will also decrease predictive power of the model. Thus the model with noise variables will have higher $R^2$ and higher AIC.

To understand more with respect to the above explanation, you can go through this Link

I have also pasted the R2 and MSE values. As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?

To answer this I think you need to derive AIC and BIC values and finally you can decide on which model to choose.

To derive the AIC value for Linear Regression can be done by using OLS, as the value of AIC is directly available.

If you want to derive, you can go through this Link

AIC:

$AIC= 2k - 2ln(sse)$

where k= number of variables

BIC:

$BIC = n * ln(sse/n) + k * ln(n)$

Do let me know if have any additional questions.