3
$\begingroup$

I am reading these two pages: xgboost documentation Post on evaluation metrics

I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large spenders, outliers. So, we care about them. I am using XGBoost.

I have tried xgboost with objective reg:squarederror. This tended to underpredict a little. I then tried with reg:squaredlogerror and this resulted in predictions that under predict by much more than just using squarederror.

I have tried tuning with several differing hyper parameter combinations but none made as big a difference as changing the objective. So, I'm dwelling on the objective function and trying to understand if there's another one out there that would be worth a shot?

On the xgboost docs above, some of the out of the other regression objective options are reg:pseudohubererror as well as count:poisson.

There is no option, that I can see, for just MAE. If using an objective function less susceptible to outliers with rmsle took me further away from accuracy whereas rmse took me closer, would using MAE potentially be worth a shot? In this dataset, outliers are more important, but so are regular users.

What would be a good objective and evaluation metric? Is MAE worth trying? If so, how? Looking at the docs above, I cannot see MAE as an option under regression parameters.

$\endgroup$
  • 1
    $\begingroup$ With MAE it will be even less sensitive to outliers than with MSE $\endgroup$ – David Masip Jul 6 at 16:46
  • $\begingroup$ In that case, more in the direction of rmsle which took me further away? $\endgroup$ – Doug Fir Jul 6 at 16:53
  • $\begingroup$ Asking my question in another way, is there an objective function more sensitive to outliers than rmse? $\endgroup$ – Doug Fir Jul 6 at 17:02
  • $\begingroup$ Given what you've shared, I'd try to improve the model (feature selection/transformation, regularization if overfitting is your problem, etc), not the loss function. MSE sounds good for your issue. $\endgroup$ – Itamar Mushkin Jul 7 at 13:34
2
$\begingroup$

These are several things you can try:

  • Use quartic error, $(y - \hat{y})^4$, instead of quadratic error. This is going to penalize a lot big errors, way more than MSE. The issue is that this is not implemented in xgboost, and you would need to develop a custom loss.
  • If your target is always positive, you can use the target as training weights. This will give more weights to the outliers. If it is not always positive, you can use the absolute value of the target as weights. If using the target values directly puts too much weight on the outliers, you might want to transform it (e.g. using the log or square root), and if you have samples whose target value is zero, you might want to add some epsilon to all the weights. Note that xgboost can be easily trained using weights.
  • Try to predict the quantile of the training distribution, then transform your predictions using the training cumulative probability function.
| improve this answer | |
$\endgroup$
  • $\begingroup$ +1 for using weights for the 'big spenders'. That sounds like a good idea. I'd also suggest exploring transformations of the target, like taking the square root or the log (if it's spending, it's never negative, right?), or adding some epsilon to avoid dropping users with zero spending from the dataset. $\endgroup$ – Itamar Mushkin Jul 7 at 13:32
  • $\begingroup$ yeah, I have tried myself at some point on a similar problem but didn't work very well. still, it needs to be tried. I'll include that $\endgroup$ – David Masip Jul 7 at 13:56
  • $\begingroup$ This is great, I have several new things to try out now. Thank you for the tips. I'll update here if I am able to make any headway $\endgroup$ – Doug Fir Jul 7 at 16:22
  • $\begingroup$ @DavidMasip I've suggested an edit $\endgroup$ – Itamar Mushkin Jul 8 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.