Most discussions on model prediction says that you should focus on error metrics, like RMSE, MSE, MAE or MAPE. Some even argue that r-squared can be low in a good model. However, I can't think of a model that would have a low r-squared and "good" error metrics. Is this possible? In which situation?
2
-
1$\begingroup$ $R^2$ is a function of (R)MSE that compares the MSE to a benchmark value, so the idea of looking at the (R)MSE instead of the $R^2$ has issues. What sources do you have advocating for this? I would be interested in the exact phrasing. $\endgroup$– DaveCommented Jul 9 at 17:37
-
$\begingroup$ define good model... $\endgroup$– Alberto SinigagliaCommented Jul 9 at 23:07
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
$R^2$ is a function of the (R)MSE that compares the MSE to a benchmark value coming from predicting the mean every time, regardless of feature values.
$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) = 1-\left(\dfrac{ N\times MSE }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)= 1-\left(\dfrac{ N\times \left(RMSE\right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$
If you have a low $R^2$ value, it means that your (R)MSE is not much lower that the (R)MSE of that benchmark model that always predicts the mean. It might be that you don't have to be much better than that benchmark, however. For instance, if you can do just a little bit better than an investing benchmark, you might be in a position to make a ton of money despite having an $R^2$ or $R^2$-type of value that is quite low.
