0
$\begingroup$

I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of the model. The average RMSE and MAPE of my model seem too high, with values of 3.4 million USD and 515 % respectively. However, looking at the RMSE and MAPE distribution in relation to the prices, I can see that there are a few, very low prices, that are difficult to predict and cause the very high mean error estimate. The scatterplot below shows price on the x axis and MAPE on the y axis

enter image description here

If I eliminate those 9 'outliers', the RMSE and MAPE go down to more reasonable values, namely 0.95 million USD and 26.7% respectively. My questions are:

  • should I eliminate those 'outliers' (i.e. very low prices) before training?
  • or should I train the model to predict all the data, and then exclude the outliers from the final RMSE/MAPE estimation?
  • or is there any other best practice in providing the most realistic estimation of the error
  • or, is there any other error metric that is robust to outliers?

Thank you very much

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In general you should not attemt to make your model look better than it actually is. Always stick to the truth. Some ideas:

  • Outliers: It is not okay to simply drop the outliers. If you want to make a prediction or do inference, you have to ask yourself how relevant such outliers are in practice. Where do they come from (are data reliable or is it an error)? If they are part of the true bigger picture, you cheat if you just drop them. If they are a very special part of the market with specific attributes, you could try to control for these special types (e.g. by adding indicators or changing the parametric form of your model). This question essentially is data and method driven.
  • Your metrics should of course reflect the final model!
  • By definition of the RMSE it is the mean (not median). Naturally, you can describe the situation by discussing the influence of the outliers in your own way. But this will not make your model/fit better after all.

Finally: I think you should NOT ask about a metric that suits your approach and make it look better. You should ask yourself if you can improve your model. You did not specify what type of method you use and what data you work with, but I'm quite sure there are ways to improve the fit. The scatter plot looks like an L-shaped function. So this might be the first thing to start to work on!?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi Peter, thank you for your answer. To clarify, I was not attempting to make the model look better, but to give a realistic and useful estimation of the error. If I say that the MAPE is 500% nobody will use the model, while it could in reality be useful except for very small orders (with low prices). Due to the nature of the exercise, I can't know if those low prices are genuine or mistakes in the data. In terms of the L-shaped function you mentioned, what does this entail? Please note the plot is of the error vs price $\endgroup$
    – LifLif
    Jun 12, 2019 at 11:45
  • 1
    $\begingroup$ As for the models I used, I have compared KNN regression, Extra Trees, XGBoost and SVR, with XGBoost being the one to return the lowest RMSE and lowest Standard Deviation in cross validation. Any suggestion for a better approach? $\endgroup$
    – LifLif
    Jun 12, 2019 at 11:46
  • $\begingroup$ Right: maybe two thing... a) (as suggested above) try to find anything that the outliers have in common to add additional controls/features. b) I think boosting is the way to go. I'm not a great fan of xgboost (opinion of course). I have very good experience with lightGBM. It may be that even a linear regression with smoothing splines gives a good result (because of the non-linear pattern in the data). stackoverflow.com/questions/38450396/… $\endgroup$
    – Peter
    Jun 12, 2019 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.