How do you identify whether your RMSE score is good or not?

Question

Im building a XGBoost regression model to predict the values in the range of -3 to 3. Im using Root Mean Squared Error to evaluate the model. With hyper-parameter tuning and everything the best scores I could get was this:

1. training - 0.118
2. validation - 0.3207 (5 folds)
3. testing - 0.3018

I know for the regression task, the RMSE values should be as less as possible. But however, I couldn't reach the range of 0.2 atleast for the test score.

Im wondering does my validation and test score range of 0.3 has anything to do with my -3 to 3 continuous target range?

Any quick help here is much appreciated. Thank you

Answer 1

Note that RMSE is an easy to understand metric. Its the Root of the Mean Squared Error. So this is just how is the typical error.

If your target is something like how big is a building, and the mean of the target its 100m, then having an error of 0.3m its nothing. On the other hand if you predict the size of insect, and your target mean is around 0.1m then an error of 0.3m is huge.

For your case it seems like a good result, an RMSE in test of 0.3, in a range of [-3,3], but then this depends on your problem.