Determining which model result is better

Question

I am trying to determine which model result is better. Both results are trying to achieve the same objective, the only difference is the exact data that is being used. I used random forest, xgboost, and elastic net for regression. Here is one of the results that has low rmse but not so good r2

model   n_rows_test n_rows_train    r2                rmse
rf      128144      384429        0.258415240861579  8.44255341472637
xgb     128144      384429        0.103772500839367  9.28116624462333
e-net   128144      384429        0.062460300392487  9.49266713837073

The other model run has a higher r2 but not so good rmse relative to the standard deviation.

n_rows_train  n_rows_test     metric_col        model   rmse           r2
37500         12500        3 year appreciation  e-net   62.3613393228877    0.705221446139843
37500         12500        3 year appreciation  rf      52.0034451171835    0.795011617995982
37500         12500        3 year appreciation  sgd     1952637950501.17    -2.89007070463773E+020
37500         12500        3 year appreciation  xgb     50.3263561914699    0.808019998691306

Which one is better?

Answer 1

Another way to approach the problem is to take all of the trained models and compare each of their performances on the same hold-out dataset. This is the most common way to evaluate machine learning models.

Choosing the evaluation metric to use depends on the goal of the project. Most machine learning projects care about predictive ability.

R² is not a useful metric for the predictive ability of a model.

RMSE can be a useful metric of predictive ability. However, since the errors are squared is sensitive to the properties of the data. You mention that you are using different data. Those differences in data could impact comparing RMSE across different sources. Comparing different models on the same dataset would be better when using RMSE.

Answer 2

One interesting note, before proceeding:

The $RMSE$ and $R^2$ values for your problem have a strong negative correlation. Just look at this graph with the first three values:

The correlation coefficient is coming out to be -0.9999525 considering the first three models only. Now if, others are included, then also it stays the same.

Now coming to your answer, $R^2$ tells us about the % of variance that can be explained by the model. Whereas, $RMSE$ can tell us about how close our predictions were compared to the actual values. $RMSE$ often gives a better estimate about the model but it suffers badly when there are outliers in the prediction.

So, for that reason, I would suggest the following step before taking the final decision:

Compute the $MAD$ (Median absolute deviation) and $MAPE$ (Mean absolute percentage error) and see how are they coming. $MAPE$ also suffers if there are outliers in the model but we can get a better sense of the prediction as it takes the relative %.

In short: I would look at more Statistical Error Estimators before selecting the best model.

In addition, have a look at this paper, which I co-authored:

Prediction of Responses in a Sustainable Dry Turning Operation: A Comparative Analysis

Here, we have used 5 different statistical error estimators to select the best model.

Answer 3

The model result with higher R2, where the R2 score move towards one, the model performance improves. You must note that R2 score is a metric that tells the performance of your model, not the loss in an absolute sense that how well your model performed.

In contrast, MAE, RMSE and MSE depend on the context whereas the R2 score is independent of context.

For these reasons second model result is better.