# Using neural network regression for skewed data

I'm using a feed forward network for a regression problem where the response variable is a ratio that can be negative and is very heavily skewed. As the response can be negative, I can't just log transform the raw data. I've thought about doing log(1+x), rank(x) and standardize(rank(x)). What would be the best way to deal with this while preserving as much feature of the data as possible?

• Could you just add a constant value to your input data, like 1 or 10 here, so all values are positive?
– KAE
Oct 2, 2018 at 19:33
• Nor-normal response variables (all data pooled together) aren't even an issue for vanilla OLS. (Consider ANOVA: if the data are separable into multiple groups, your pooled response variable is some kind of Gaussian mixture and definitely is not normal.) What's the problem for a neural net? If it really is a problem, however, what about the logistic function?
– Dave
Dec 12, 2019 at 12:42

I faced this problem many times, these are the two solutions that worked best based on my experience:

1. The log(x+1) transformation as you proposed: you take log( x + 1 ). This is the log of the distribution. If your dependent variable has a lower bound (i.e. a minimum value it can possibly take) then I suggest you to use log( x + 1 ) + min( x ). In this way you constrain the distribution to take only positive values, and use ReLU at the output node

2. One that I tried recently at work and that worked very well is a scaling between 1 and the 95th percentile. This operation is analogous to the more classical min-max scaling, but instead calculating it using min(x) and max(x), you shring min(x) and 95th percentile. I said 95th percentile, but 99th or any other could also work well - you can choose the one that best fits your task. This become very useful when simple min-max scaling doesn't work (i.e. when shrinking everything in the 0-1 range would cause smaller observations to "disappear"). What you'll have in the end is a distribution with almost any observation in the 0-1 range, with very few outliers beyond that, with little influence on the overall quality of the prediction. This technique assume a ReLU activation at the output node. In this way you make sure to always get outputs in the desired, meaningful range.

I tried other techniques (one based on the upper part of Sigmoid function, and many other based on sequences of transformations with logs, square roots, standardization, and other monotonic functions), but they didn't perform better than the two I chose. Of course this is only based on personal experience.

PS: These custom standardization cannot rely on methods such as sklearn's .fit() method. Remember to manually save the parameters in a specific file, otherwise you won't be able to revert your distribution back to the original!

For future use:

I encountered a similar problem. In my case neural network failed to predict accurately in skewed part of the data. I created bins (10 in my case) and gave weights to data. After than I used important sampling in batch gradient descent. This can be easily done using keras generators.

Below is the generator function.

def generator(features, labels, batch_size, w):
n = features.shape
w = w/w.sum()
while True:
ind = np.random.choice(n,batch_size,p=w)
batch_features = features[ind]
batch_labels = labels[ind]
yield batch_features, batch_labels


This generator samples skewed data more often than random sampling and makes sure the model is not biased towards majority data cloud.

w is weights. Here is the brute force code to calculate weights. I am sure there are libraries to calculate this.

data['bins'] = pd.cut(data['response'], [0,1,2,3,4,5,6,7,8,9,10], labels=[1,2,3,4,5,6,7,8,9,10])
weights = data['bins'].value_counts()
weights = 1/weights
weights = weights/sum(weights)
weights = weights.to_dict()
for i in data.index:
data.loc[i,'weights'] = weights[data.loc[i,'bins']]

• Thanks for this. What did you use for w? I presume you'd want to sample the skewed part more heavily? Dec 11, 2019 at 2:08