Regression: How to deal with positive skewness in continuous target variable

Question

I'm working on a regression problem. My aim is to "learn" the distribution of a continuous target $y$ as good as possible to make predictions. My model looks like:

$$y_i=\beta X_i + u_i.$$

• $y$ is right skewed (positive skewness) and consists of positive and negative integer values.

• $X$ is a matrix containing columns with float and integer values. There also is a large(er) number of indicators (dummy variables)

These are my results. Here I compare the distribution of the actual $y_i$ (truth, blue) and the predicted $\hat{y_i}$ (predicted, yellow):

As you can see from the figure, I tend to underestimate values in the lower range (around zero) and I overestimate values somewhere between 200 and 500.

I would like to improve my estimation especially in the range somewhere around zero and 600, where most observations are found. I wonder what my options are?

Please note that the whole range of data is relevant to me, so that I cannot simply prune the data/distribution.

So far I did no feature engineering, scaling etc. What I tried was a number of different estimation methods, such as linear regression (OLS), generalised models (GAM) with local regression and regression splines, lasso/ridge, Keras neural net, boosting with LightGBM and Catboost. All of them produced very similar results (as shown in the figure above).

Edit [2019-12-28]:

A brief follow-up to my original question: I have applied a logistic transformation to my $y$ and estimated a GAM model. Unfortunately the results are not much better than with non-transformaed data. MAE decreases slightly, but the general fit is still not too good (see figure below, the "flat" curve are actual values and the pointed curve are predicted values after re-transformation).

I prefer to stick to linear models (OLS, GAM, Lasso) since boosting or neural nets did not deliver much better results than linear models.

Answer 1

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions.

Some takeways:

• Add count of values of features as new features. This is information that LightGBM can’t see
• Check unique value counts for all features of train and test set.
• Model stacking almost always works better than single model (forget about it in production)
• Check for feature independence. For this competition, features were all uncorrelated (naive bayes worked great)
• etc...

And finally and independently from this competition general note: Its all about the right data representation and not about the model. If you can represent data in the right way than basic linear regression will score good. I would do some serios data analysis for residual values in range negative until 200 and 500 until infty and see what confuses your model, obviously there are some conflicting features that force your model to predict low when it should be high. Maybe this is not satisfying to you but to really solve this problem you have to find what confuses him so that he predicts high when it should be high, and then engineer feature/data representation so that he can discriminate around this.

Answer 2

You could apply a square or a square root transformation to the y and check for which transformation y is not skewed.

Answer 3

Solutions to over fitting:

• Simplify model by reducing the number of attributes in the training data or by constraining the model(regularization).
• Gather more training data
• Reduce the noise in the data (Fix data errors, remove outliers)

Solutions to under fitting:

• Feature engineering
• Reducing the constraints in the model.

To make good predictions your training data must contain enough relevant features, good quality data (free from outliers,noise and errors).

Answer 4

for right-skewed (positive skew) data log-transformation could be used:

df['bin'].apply(np.log).hist(bins=20)
plt.xlabel('log_bin')
plt.ylabel('frequency')

• in order to get Normal distribution

p.s. and if left-skewed: use a generic way to transform non-normal distribution: the Box-Cox transformation.