1
$\begingroup$

I have regression model, where target is between 0 to 1. standard deviation of target is 0.817 and RMSE of model on hold out is 0.52. I am wondering if this good model or not.

Any feedback will be useful

Thanks

$\endgroup$
1
  • $\begingroup$ it is definitely not good, both the mean error and the spread are high. I would consider it good if both measures were around 0.1. But the answer also depends on your application. $\endgroup$
    – pcko1
    Jul 5, 2018 at 17:11

2 Answers 2

1
$\begingroup$

The residuals of your model seem to vary by a large extent (as denoted by RMSE, which is in the same units as the target). Without any context on the problem you are solving, it would be hard to decide whether the model is good or not. But, it does appear to be on the poorer side.

$\endgroup$
0
$\begingroup$

I will assume a linear regression model fitted with ordinary least squares. If this is incorrect, then what I am saying about $R^2$ does not quite apply; in other cases, $R^2$ is not equal to the proportion of variance explained.

If your standard deviation of all $y$ observations is $0.817$, then the variance of $y$ is $0.667489$. If the $RMSE$ is $0.52$, then the variance of the residuals is $0.2704$; the is the mean squared error, $MSE$.

$$R^2 = 1-\dfrac{SSRes}{SSTotal} \approx 1-\dfrac{nMSE}{nVar(y)} = 1-\dfrac{MSE}{Var(y)}= 1-\dfrac{0.2704}{0.667489} = 0.5949 $$

You explain almost $60\%$ of the variability of $y$.

(There's this issue of $n-1$ vs $n$ vs $n-p$ in the denominators that makes this not quite true. Either calculate the MSE and $var(y)$ with the same denominator, or assume that you have enough observations to wash out this effect, which you might have.)

The trouble that I see with $R^2$ is that people want to compare it to grades in school, where $R^2=0.95$ is like $95\%$ and an $A$ that makes us happy, and $R^2 =0.6$ is like $60\%$ and an $F$ that makes us sad. There could be cases where $R^2=0.5$ is suspiciously high, and there could be cases where $R^2=0.95$ is pedestrian. (Think about how an accuracy of $95\%$ with the MNIST digits is far from remarkable performance.) You might see your regression as being equivalent to a failing grade in school; it see it as quite the improvement upon how one could predict $y$, but perhaps not better than existing methods. (What if others tend to get $R^2 > 0.7?$ Then your model might not be good enough unless it can make up for the decrease in performance by, for instance, being cheaper to implement (perhaps it is cheaper to measure your variables.)

My preference is to put the performance in the context of how the model would be used and how it improves upon existing methods. Depending on the performance metric(s) you use, this avoids the issue of nonlinear regression not having the convenient sum of squares decomposition that OLS has.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.