High root mean squared error in regression model

Question

I am applying regression to a dataset comprising 110 rows and 7 columns each with targets. When I applied Lasso regression to the data and calculated the RMSE value, the RMSE value was 13.11. I think the RMSE value should be close to zero. What is the permissible values of RMSE in a regression model? What could have gone wrong in the computation?

My code:

from sklearn import linear_model
reg = linear_model.Lasso(alpha = .00001)
reg.fit(Xt,Yt)
ans=reg.predict(Xts)
print(ans)
from sklearn.metrics import mean_squared_error
print(mean_squared_error(Yts, ans))

Whereas when I try cross validation the MSE scores are way below 0.35

kfold = KFold(n_splits=10)
results = cross_val_score(reg, full_data, target, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
results
Results: -0.13 (0.45) MSE

Answer 1

RMSE does not work that way. A RMSE of 13 might actually be great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range [0,1e9], then a RMSE of 13 is spectacular. On the other hand, if your target is in the range [0,1], a RMSE of 0.5 is terrible. If you would like to try a metric that can be more readily interpretable as having a "good" or "bad" score, try the Mean Average Percent Error (MAPE).

As far as why you get a lower MSE when you cross validate: you do not show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I would recommend using bootstrap cross validation instead of k-fold.