2
$\begingroup$

I am applying regression to a data of 110 rows and 7 columns ,each having targets. When I applied Lasso to the data and calculated the RMSE value ,the RMSE value is coming to be 13.11.I think the RMSE value should be close to zero.What is the permissible values of RMSE in a regression model.What could have gone wrong in the computation.

from sklearn import linear_model
reg = linear_model.Lasso(alpha = .00001)
reg.fit(Xt,Yt)
ans=reg.predict(Xts)
print(ans)
from sklearn.metrics import mean_squared_error
print(mean_squared_error(Yts, ans))

whereas when I try cross validation the MSE scores are way below 0.35

kfold = KFold(n_splits=10)
results = cross_val_score(reg, full_data, target, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
results
Results: -0.13 (0.45) MSE
$\endgroup$
  • $\begingroup$ Apply grid search for your alpha value and try adding more features and make sure that the problem is solvable by a linear model as it will die lower your alpha value, higher the model will be punishing incorrect classification (let say) $\endgroup$ – Aditya Mar 20 '18 at 7:59
5
$\begingroup$

RMSE doesn't work that way. An RMSE of 13 might actually great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range [0,1e9], than an RMSE of 13 is spectacular. On the other hand, if your target is in the range [0,1], an RMSE of 0.5 is terrible. If you want to try a metric that can be more readily interpretable as having a "good" or "bad" score, try Mean Average Percent Error (MAPE).

As far as why you get a lower MSE when you cross validate: you don't show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I'd recommend using bootstrap cross validation instead of k-fold.

$\endgroup$
  • $\begingroup$ My values are in range 0 to 95 $\endgroup$ – Boris Mar 20 '18 at 8:45
  • $\begingroup$ Then scale them a bit $\endgroup$ – Aditya Mar 20 '18 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.