4
$\begingroup$

I am working on Decision Tree algorithm and at the end I calculate RMSE value based on actual labels and predicted values (for regression). Now what I am having difficulty in is in understanding the significance of the RMSE value that I get. I know that the lower the RMSE better is the performance of the model but what RMSE value is considered reasonable low or high? Suppose my RMSE value comes as 20 so does that mean the performance of my model is 80% (means 80% of predictions are correct)?

$\endgroup$
2
  • $\begingroup$ A similar question $\endgroup$ – Dawny33 Dec 2 '15 at 14:40
  • $\begingroup$ SO, Standard deviation of what? Observed value or Predicted value? $\endgroup$ – Kaushik Sep 9 '17 at 9:04
3
$\begingroup$

There are multiple factors to consider, but the first thing to realize is that in regression, you don't want to think about whether an example is "correct" or "incorrect" but rather how close it was to the true target value. Therefore you can ignore your original intuition about "80% of predictions are 'correct'."

Second remember that RMSE is in the same space as your target values. So it is relative to the variance in your target values. The benchmark of random guessing should get you an RMSE = standard_deviation. So lower than this, your model is demonstrating some ability to learn; above that number, you haven't even learned to guess the mean correctly.

There isn't a cutoff for "my model is doing well" in RMSE space, just like with other metrics. Everything is relative to a naive solution/benchmark or the state-of-the-art.

$\endgroup$
2
  • $\begingroup$ So in order to check if my RMSE is good or bad, I can compare it with the standard_deviation to get a rough idea? The standard_deviation should be calculated in actual label values? How can I calculate the standard_deviation? $\endgroup$ – Jason Donnald Dec 2 '15 at 23:07
  • $\begingroup$ Depends on the package you're using, but I'm certain every toolkit for ML will have an easy stdev function. It's also easy to calculate yourself. Basically the StDev is a rough idea of if you're learning anything at all. If you want a better idea, compare to other models. $\endgroup$ – jamesmf Dec 2 '15 at 23:14
2
$\begingroup$

The root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model and the values actually observed. The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you've badly over fit the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.