I am working on a counterfeit medicine sales prediction regression model. As the relationship between target & response variables is non-linear I used tree based regressors random forests and XGB. The mse on validation data for both RF & XGB are similar but since XGB overfitted less I went ahead with it.

Then I calculated permutation feature importances & selected the important features.

The problem in the model is it is consistently under predicting high values of target variables as shown in the image.

Actuals vs Predictions-XGB

Below is the distribution of target variable/sales.It is quite skewed but am not sure if that's the reason for bad predictions for high sales values. Can anybody please guide me how deal with this? Do I need to apply some multivariate outlier removal technique or is there any other solution to this.

count 6814.000000

mean 2280.450245

std 1693.360070

min 146.290000

50% 1902.670400

80% 3540.804720

90% 4644.967440

95% 5582.547000

97% 6155.135000

98% 6679.638924

99% 7351.238042

99.5% 8246.805622

max 13199.964800


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.