Boosted tree regression loss function when data has occasionally very large values to predict?

Question

I have a regression problem where most of my target variables are down in the range 5-30, but occasionally the target variable will spike up to 100, 500, or even 5000. These values are not spurious outliers that should be removed, but are values I'd like the prediction algorithm to try to capture. However, I do not want the error on these variables to dominate the training of the tree. Conceptually, the percent error is more akin to what I'm interested in (although it doesn't have to be it exactly). Specifically, when the target is 30 and I predict 15, I consider that just to be similar as when the target is 5000 but my prediction is 2500. I don't want a 2500**2 squared error to overwhelm the 15***2 squared error.

For this type of problem, what is the best way for me to tackle this issue? Data transform? Custom loss function? Etc?

Answer 1

I am working almost on the same problem these days:

I have tried two options using XGB Regression with different objective functions including:

1. Using a linear regression objectiive function ("reg:linear" or "reg:squarederror") and transformed the target to the log space

2. Using the gamma objective function ("reg:gamma"), which is useful for a skewed target with gamma distribution, e.g., insurance claims severity. In this case, I did not transform my target to the log space.

You can try these two cases and see which one performs better. However, in my case, option 1 performed better than option 2 (around 15-20%).

Also, you can try "reg:squaredlogerror"

Answer 2

Flier values/skewed predictors will have a high influence on the regression model. If you want to counteract that, you have a few choices.

1) If your target is always non-zero, and if you expect the regression to be close to linear, you can try to use a log(), sqrt() or even boxcox() conversion transform on the target variable. This will help keep the large values from having a big influence. Also, if you are normalizing the data, you should run the transform first. Just remember to convert the prediction back using the exponential function. You can check if the skew has decreased using the skew() function (lower is better)

2) You can add a weight value or loss/cost function. Here is a good reference for these options:

https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression.html#initialize-weights