How to deal with overestimation of small values and underestimation of high values in XGBoost?

Question

I'm running XGBoost to predict prices on a cars dataset, I was wondering what alternatives are there for this kind of problem where smaller values are overestimated and higher prices underestimated.

I tried applying log to prices since it has a skewed to the right distribution, but still having this undesirable effect.

Also, as a bonus question, log(price) improved the prediction score, the mean relative error or MRE calculated as mean(ABS(RD)) by 2 percent, if anyone has the intuition onto why this could have happened that would be great.

In the image below RD is the relative difference between predictions and the actual values, and the price bucket is a bucketized variable where the number indicates the price low interval bound over 1000.

Answer 1

I dont think its necessarily related to the type of algorithm performing the regression(XGBoost here) - but to inherent nature of regression algorithms.
Many loss function are aimed to reduce distance between $y$ and $\hat{y}$.
That can lead to model predictions distribution being tighter around $y$'s mean.

Couple of things I would check to verify this:
1) Compare distribution of real $y$ and predicted $\hat{y}$.
2) Verify this error pattern on others regression model.

Things I would try to improve results:
1) over sample low/high y values in training set.
2) Adjust loss function so errors on low/high y values will have more weight.
3) Look for features that emphasize low/high y values and engineer them better.

Answer 2

Maybe you can try using multiple XgBoost models instead of 1 and take an average (or weighted average) of their predictions.