Evaluating the test set

Please find attached a part of the code which explains what I'm trying to do. Essentially I'm trying to predict the sales of supermarket stores. Im using RandomForestRegressor for this and have predicted the results on the test set. The cross validation is done on the training set with a mean accuracy of 0.52 and I have tried to calculate RMSE which comes to 1180. Now, I want to know how do I make sense of all this and know how well it performed on the test set and how to evaluate my model. It'll be extremely helpful if you could make me understand. Thanks

Ignore the np.exp(rmse) part

Best,

Pranveer

welcome to the forum, I will try to clarify some things.

First of all, when talking about regression you do not calculate accuracy. Accuracy is a metric used in classification tasks, it basically measures how much labels your model got right divided by the total labels.

You are working in a regression task, that is, you are trying to predict not labels, but continuous values as your target. By default, scikit-learn estimators calculate as regression scores the R_squared metric. Intuitively, R_squared measures how much of variance your model explains. It is calculated by the below equation:

Basically, I like to think that R2 is testing your model against the simplest possible one in a regression task, that is, just predicting the mean for your target variable. It makes sense talking about accuracy as an analogy, however I wanted to make clear that they are different things.

Second, talking about results, RMSE and how can you measure the effectiveness of your model, in linear regression context it will be highly dependent of the problem your are trying to solve. Maybe you are a social scientist trying to predict with some data the number of voters in certain candidate, an R2 of 0.4-0.5 would be amazing (?). Maybe you are sure that this is not sufficient because your variable are strongly related to your target variable (some problems and physics) and a R2 of 0.80 is unacceptable. Another thing is RMSE, it tries to measure the variance of your model that is, how far off you usually are when predicting your target. I had a problem where I was prediciting the number of days of a certain event to happen, and my variance was 15 days off, and this was not helpful at all.

A highly recommend you to read the Chapter 3 of introduction to Statistical Learning Book that will help you a lot (Actually, the whole book). They have a free version here: http://www-bcf.usc.edu/~gareth/ISL/

I hope this helps. Also, any problem to understand what I wrote, please let a comment. And if anyone spots something wrong in my answer, tell me please!