1
$\begingroup$

Having the following distributions (actual and predicted), Hist 1 to 3 (left to right). enter image description here

I would like to get a score ranging from 0-1 of how close the actual distribution is to be normal. I've found a couple of statistical normality tests:

  • Shapiro-Wilk Test
  • D’Agostino’s K^2 Test

My DataSet is large therefore I've decided to check the skew and kurtosis statistics and got the following results:

hist-1 Skewness is 0.028386209063816035 and Kurtosis is 2.4224694251429764 <-- Most normal
hist-2 Skewness is 3.7702212103585246 and Kurtosis is 15.214567975037294
hist-3 Skewness is -0.40471550878367296 and Kurtosis is 1.4106438684701157

How can I calculate a score ranging between 0-1 using those params? Or is there a better approach to calculate the score?

Update: As suggested, I've tried stats.kstest(data,"norm"), however the results does not differentiate the differences between the distributions, or maybe I'm missing something?

Hist-1 - KstestResult(statistic=0.9274310194094191, pvalue=0.0)
Hist-2 - KstestResult(statistic=0.9999966401777812, pvalue=0.0)
Hist-3 - KstestResult(statistic=0.9911610021388533, pvalue=0.0)
$\endgroup$
2
  • $\begingroup$ does it have to be between 0 and 1 ? is that something you don't want to negotiate ? :-) $\endgroup$
    – user702846
    Jun 29 '21 at 7:57
  • $\begingroup$ I just want a range that would be consistent between different sets of data $\endgroup$ Jun 29 '21 at 8:11
2
$\begingroup$

You can use the Kolmogorov-Smirnov statistic, which by construction lies in $[0,1]$ since it is the supremum of the pointwise differences between the cdfs of the two distributions compared.

Incidentally, being non parametric you can also use the same test to compare your actual and predicted distributions.

$\endgroup$
5
  • $\begingroup$ Thanks for the reply, please see my update $\endgroup$ Jun 29 '21 at 7:29
  • $\begingroup$ @Miguel does these test give a score from 0 to 1 ? do you think the p-value of 0.001 vs 0.00001 means this ?! $\endgroup$
    – user702846
    Jun 29 '21 at 7:56
  • $\begingroup$ It's the statistic's value that is between 0 and 1. High values mean great differences. Note that you are testing against a normal Gaussian, i.e. mean 0, variance 1, so you have to normalize your data before the test. About the p-values, I recommend you read this thread. $\endgroup$
    – Miguel
    Jun 29 '21 at 8:26
  • $\begingroup$ wrong - p-value is not the effect size. p-values do not show to what extend they are different $\endgroup$
    – user702846
    Jun 29 '21 at 8:30
  • $\begingroup$ Note that the answer is not suggesting to use the KS test but the KS statistic $\endgroup$
    – Miguel
    Jun 29 '21 at 18:47
1
$\begingroup$

First just using statistical test to get p-value in this context is wrong. Why ? because p-values just indicate the level of significance not the amount or the amplitude of difference. In other words one after running two statistical test can not say p-value of 0.00001 is a better fit compare to p-value of 0.001 - even though both are below the commonly accepted threshold. This means, the test with the smallest p-value has a higher chance to be significantly different than the second test, but does not tell you how different. Therefore, you always need an effect size. Again does not say the first test has a smaller error ! here you can read why p-values here are not enough. There are tons of other posts that you can find when you search for "p-value vs effect size"

Since you are after ranking them here is my suggestion:

To the get the worse histogram you generate random data

data = np.random.normal(0, 0.5, 1000)

then you fit a normal distribution

mean, var  = scipy.stats.distributions.norm.fit(data)

then you calculate the error for example MSE. Basically, pick an x from your data, find the y in the normal distribution and subtract from your own y. From here you would have your maximum level of error.

then for each of your histogram you fit a normal distribution and get an error as described above. Now your error from the fit are comparable, the histogram with the least amount of error is the best fit. You can use p-values to show that it is significant. Perhaps wilcoxon test a non-parametric test could be an option. Since you do not know your data are normal or not you can't use any test that has a normally distributed assumption.

Anyway, to bound your value to [0, 1] then you have to normalized your error. This means, your worst error should be 0 (comes from the random number) and the best 1 (a perfect match perhaps).

$\endgroup$
8
  • $\begingroup$ Note that the KS statistic is already in $[0,1]$. It's just the $\ell_\infty$ norm of the difference of the CDFs. $\endgroup$
    – Miguel
    Jun 29 '21 at 18:45
  • $\begingroup$ @Miguel - The p-value returned by the k-s test has the same interpretation as other p-values. You reject the null hypothesis that the two samples were drawn from the same distribution if the p-value is less than your significance level. I am not saying can not be used. What I am saying is one can not sort those histogram purely based on p-value. you need to calculate an effect size $\endgroup$
    – user702846
    Jun 30 '21 at 12:27
  • $\begingroup$ p-values are like probabilities - p-value 0.0001 doesn't mean the signal is stronger than p-value of 0.01. bot reject the null hypothesis - One need to measure the actual signal - again I encourage you to read "p-value vs effect-size" - statistics is much more boring than AI/machine learning !! $\endgroup$
    – user702846
    Jun 30 '21 at 12:30
  • $\begingroup$ Again: I never even mentioned p-values or the KS test, only the KS statistic! As every (?) other test, the KS test uses asymptotic assumptions on the distribution of some statistic (the KS statistic in this case) in order to build tests at different levels and from this you can build the $p$-value. $\endgroup$
    – Miguel
    Jun 30 '21 at 12:34
  • 1
    $\begingroup$ No worries! Glad it's clear now 👍 $\endgroup$
    – Miguel
    Jun 30 '21 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.