RMSE vs R-squared

Question

Question: Which is a better metric to compare different models RMSE or R-squared ?

I searched a bit usually all the blogs say both metrics explain a different idea, R-squared is a measure of how much variance is explained by the model and RMSE is giving you hint on average error.

My answer: I think RMSE can be used to compare training error and validation error basically telling if model over fits or not. This will also say how well can two models perform on unseen data but R-squared only says information about model fit it gives no information about how model will perform on unseen data.

Hence RMSE is better than R-squared if you worry about how your model will perform to unseen or test data.

Is my answer correct ?

(Note: Please add any points if you know any scenario when R-squared is better than RMSE)

Answer 1

Look at the equations. Both are functions of mean squared error. Any model the outperforms on one will outperform on the other. The danger I see with $R^2$ is that it puts us in a position of thinking of grades in school, yet an $F$-grade $R^2=0.4$ could be quite excellent for some models, while an $A$-grade $R^2=0.95$ could be quite pedestrian for some models. Further, $R^2$ loses its “proportion of variability explained” interpretation in the nonlinear case (and even some linear cases when we do something other than ordinary least squares): https://stats.stackexchange.com/questions/494274/why-does-regularization-wreck-orthogonality-of-predictions-and-residuals-in-line.

$$ SSResiduals = \sum_{i=1}^n \big( y_i - \hat y_i \big)^2\\ RMSE = \sqrt{MSE} = \sqrt{\dfrac{SSResiduals}{n}}\\ R^2 = 1 - \dfrac{SSResiduals}{SSTotal} = 1 - \dfrac{n\times MSE}{SSTotal} = 1 - \dfrac{n\times (RMSE)^2}{SSTotal} $$

($SSTotal = \sum_{i = 1}^n \big(y_i -\bar y\big)^2$ is a property of a data set, not of a model, so it is basically a scaling factor.)

Consequently, smaller $RMSE$ is synonomous with larger $R^2$. However, $RMSE$ does not trick you into thinking in terms of letter grades in school.

Answer 2

Your interpretation is correct though I wouldn't say one is 'better' than the other. They both serve different purposes.

The first metric I generally check after building my model is MAPE. So I can sense the relative error there with respect to the actual predictions. Though the problem with MAPE is, if there are few outliers in your predictions then your MAPE value will be affected. This problem exists for RMSE too and can be eradicated with RMSLE (Root Mean Squared Log Error).

The point is: every error estimator will have some positives and negatives, you need to decide the best one according to your problem statement.

Answer 3

If you calculated the RMSE on a test set, then it will be a better metric in assessing how well your model will perform in predictions for future observations, i.e. estimating accuracy on an unseen observations.

R-squared, as you stated, is the proportion on variance in your training set that's explained by your model fit. Hence, the crucial difference between the two metrics: RMSE is usually calculated on test data, while the R-squared is calculated on training data.