3
$\begingroup$

I am applying regression to a dataset comprising 110 rows and 7 columns each with targets. When I applied Lasso regression to the data and calculated the RMSE value, the RMSE value was 13.11. I think the RMSE value should be close to zero. What is the permissible values of RMSE in a regression model? What could have gone wrong in the computation?

My code:

from sklearn import linear_model
reg = linear_model.Lasso(alpha = .00001)
reg.fit(Xt,Yt)
ans=reg.predict(Xts)
print(ans)
from sklearn.metrics import mean_squared_error
print(mean_squared_error(Yts, ans))

Whereas when I try cross validation the MSE scores are way below 0.35

kfold = KFold(n_splits=10)
results = cross_val_score(reg, full_data, target, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
results
Results: -0.13 (0.45) MSE
$\endgroup$
1
  • $\begingroup$ Apply grid search for your alpha value and try adding more features and make sure that the problem is solvable by a linear model as it will die lower your alpha value, higher the model will be punishing incorrect classification (let say) $\endgroup$
    – Aditya
    Mar 20, 2018 at 7:59

1 Answer 1

8
$\begingroup$

RMSE does not work that way. A RMSE of 13 might actually be great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range [0,1e9], then a RMSE of 13 is spectacular. On the other hand, if your target is in the range [0,1], a RMSE of 0.5 is terrible. If you would like to try a metric that can be more readily interpretable as having a "good" or "bad" score, try the Mean Average Percent Error (MAPE).

As far as why you get a lower MSE when you cross validate: you do not show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I would recommend using bootstrap cross validation instead of k-fold.

$\endgroup$
2
  • $\begingroup$ My values are in range 0 to 95 $\endgroup$
    – Boris
    Mar 20, 2018 at 8:45
  • $\begingroup$ Then scale them a bit $\endgroup$
    – Aditya
    Mar 20, 2018 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.