4
$\begingroup$

I have a skewed distribution that looks like this:

picture of a skewed distribution

How can I transform it to a Gaussian distribution? The values represent ranks, so modifying the values does not cause information loss as long as the order of values remains the same. I'm doing this to experiment if different distributions change the behavior of my ML models.

I'm working with Python/NumPy/Pandas/scikit-learn.

Edit: I should clarify that I have a lot of features and I'm looking to automatically transform all feature distributions. I was able to find a reasonable transformation for a single feature with a lot of experimentation, but it doesn't generalize to other features:

normalize(np.log(0.30 + original)).

** here would be image i.stack.imgur.com/uzorK.jpg but I don't have enough rep to post more than 2 images **

normalize(np.log(0.17 + another_feature_distribution)).

enter image description here

In this image the purple bars represent the original distribution of another feature, green bars represent the transformed distribution. No matter how much I tweak the constant, I don't get the high green bar on the left extreme to disappear. Also, I don't have time to manually find a formula for each feature. Not sure if these are bell-shaped enough anyway?

|improve this question
$\endgroup$
  • $\begingroup$ Based on your objective of how different distributions change the maximum likelihood estimates, should you be changing distributions from which samples are taken rather than changing a specific dataset? $\endgroup$ – JimB Sep 26 '16 at 2:47
1
$\begingroup$

You can do a log transformation on your data with the help of numpy log functionality as shown below :

log_data = np.log(data)

This will transform the data into a normal distribution. Moreover, you can also try Box-Cox transformation which calculates the best power transformation of the data that reduces skewness although a simpler approach which can work in most cases would be applying the natural logarithm. More details about Box-Cox transformation can be found here and here

|improve this answer
$\endgroup$
  • $\begingroup$ Maybe there are some extreme outliers in your data set. Try removing those and check what happens. $\endgroup$ – enterML Sep 25 '16 at 22:50
  • $\begingroup$ Wait, why would taking the log transform data from an arbitrary distribution to be normally distributed? $\endgroup$ – Matthew Drury Feb 22 '19 at 3:09
1
$\begingroup$

For contemporary viewers, an update in scikit-learn now includes the PowerTransformation in the API, providing a neat way of including these transforms in the workflow. See https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing-transformer.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.