2
$\begingroup$

For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better interpret the feature (named ipc here).

Distribution of ipc feature

The regular log transformation does not work here because of the (x-axis) spread.

How can I transform this data to a normal distribution?

A related answer has been pointed out in the comment but I am looking for some Python code excerpt as well.

Thanks

$\endgroup$
  • $\begingroup$ If those ipc values are discrete then it cannot be done - at least not deterministically. If they are continuous, and you can assume a p.d.f., then it would be possible. Are they discrete (it looks so on the graph)? $\endgroup$ – Neil Slater May 12 '17 at 23:03
  • $\begingroup$ I have another metric which is continuous and has an exponential distribution. So how can I convert for that metric? $\endgroup$ – sandyp May 12 '17 at 23:06
  • 1
    $\begingroup$ Related, on Cross Validated: stats.stackexchange.com/questions/154396/… $\endgroup$ – David Z May 12 '17 at 23:15
  • $\begingroup$ @DavidZ Thanks, I actually saw that but was hoping for some code to achieve that. I should probably highlight this requirement in my question as well. $\endgroup$ – sandyp May 13 '17 at 0:00
  • 2
    $\begingroup$ Write it yourself. The inverse normal CDF in that recipe is accessible through what scipy calls the percent point function (ppf). $\endgroup$ – Emre May 13 '17 at 0:08
2
$\begingroup$

You can use sklearn.preprocessing.QuantileTransformer (or sklearn.preprocessing.PowerTransformer) which does exactly what you want:

from sklearn.preprocessing import QuantileTransformer
import numpy as np

ey = np.random.exponential(size=100)
qt = QuantileTransformer(output_distribution='normal')
no = qt.fit_transform(ey.reshape(-1, 1))

You can plot histograms to compare "before" vs "after":

# Plot histograms to see before vs after.
import matplotlib.pyplot as plt
%matplotlib inline 
plt.subplot(2, 2, 1)
plt.hist(ey, bins='auto')
plt.subplot(2, 2, 2)
plt.hist(no, bins='auto')
plt.show()

enter image description here

The advantage of this approach is that it will also work for other input distributions, not only exponential.

$\endgroup$
0
$\begingroup$

The following code works:

import scipy
import numpy as np

ey = np.random.exponential(size=100)
cdfy = scipy.stats.expon.cdf(np.sort(ey))
invcdf = scipy.stats.norm.ppf(cdfy) # a normal distribution

Hope this helps

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.