How to handle a regression problem with skewed target and only few high values?

Question

I'm currently tackling a regression problem with skewed target variable (presented below).

Naturally, my first idea was to transform the target with natural logarithm as it'll probably help both linear regression or decision-tree-based algorithms. The second idea is to prepare a validation scheme similar to stratified k-fold cross-validation with target binned into n groups. However, my concern is that I have only few highest values:

Therefore, my test set and all validation sets error are highly dependent if one of these 4 extreme values are drawed placed within them or not. That makes it hard to obtain reliable true error estimate.

Is there anything more I can do to handle that issue?

Answer 1

You can generate symmetrical distribution(s) by suitable transformation: Your distribution is mainly right-skewed, therefore a log10 transformation is required.

You can also use an auto-binning method and combine it with dummy variables for the spikes.

If you then perform a feature selection, the learner will automatically choose the most significant features. Learners like XGBoost automatically take care of multicollinearity.