0
$\begingroup$

I am doing some data analysis, and I am using the cramer von mises test to check if two samples are derived from the same distribution. I am using the cramervonmises_2samp implementation from scipy.stats. Originally, the samples that I was using (denoted sample1 and sample2) were 800K and 200K observations. When I tried to compute this, it took quite a bit, and it ate up all my ram (32GB), causing my jupyter notebook kernel to die.

My statistical test function as defined as:

def compare_distributions(sample1, sample2, ALPHA=.01):

    res = cramervonmises_2samp(sample1, sample2, method='exact')
    
    print(f'sample1 shape: {sample1.shape}')
    print(f'sample2 shape: {sample2.shape}')

    print(f'statistic: {res.statistic}')
    print(f'p-value: {res.pvalue}')

    if res.pvalue < ALPHA:
        print('The null hypothesis is rejected. \nThe samples do not have the same distribution.\n\n')
    else:
        print('The null hypothesis is not rejected. \nThe samples are drawn from the same distribution.\n\n')
    

As an experiment to see how much data my machine can handle I decided to do the test with 50, 70, 100, 130 and 200 observations in each sample, with the samples being from scipy.stats.gamma and scipy.stats.expon respectively. The results are as follows:

sample1 shape: (50,)
sample2 shape: (50,)
statistic: 1.8978000000000002
p-value: 1.5142491077016167e-05
The null hypothesis is rejected. 
The samples do not have the same distribution.


Runtime: 6.093s

----------------------------------------------

sample1 shape: (70,)
sample2 shape: (70,)
statistic: 2.2905102040816345
p-value: 1.9285375980904996e-06
The null hypothesis is rejected. 
The samples do not have the same distribution.


Runtime: 33.788s

----------------------------------------------

sample1 shape: (100,)
sample2 shape: (100,)
statistic: 3.0496999999999943
p-value: 3.495231315392338e-08
The null hypothesis is rejected. 
The samples do not have the same distribution.


Runtime: 199.694s

----------------------------------------------

sample1 shape: (130,)
sample2 shape: (130,)
statistic: 5.4970118343195296
p-value: 5.803075867500237e-14
The null hypothesis is rejected. 
The samples do not have the same distribution.


Runtime: 783.185s

When I tried to do the test with 200 observations in each sample, this time, my laptop crashed and was in need of a restart.

My questions are:

  • why does this test require so much resources? (in terms of ram and runtime)
  • is hypothesis testing always so resource intensive?
$\endgroup$

1 Answer 1

0
$\begingroup$

This is because of the fact you are trying to calculate the exact p-value by using method='exact'. Try using method='asymptotic', as that should speed up the calculations quite a bit (2ms vs 9.2s for 50 samples on my computer) with what I suspect little to no difference in the calculated p-value.

$\endgroup$
2
  • $\begingroup$ well yes, but, how do I know how close is the asymptotic method to the exact one? And judging by the cumulative distribution function plot on some of the other samples, they look nearly identical, yet the asymptotic calculation rejects the null hypothesis. @Oxbowerce do you know how method='asymtotic' actually calculates the result? $\endgroup$
    – ptushev
    Commented Apr 12, 2023 at 18:52
  • $\begingroup$ For the actual calculation when using the asymptotic method you can look at the source code as well as the original paper. Table 7 from the original paper seems to give some indication on the difference in the level of significance when using the limiting distribution. $\endgroup$
    – Oxbowerce
    Commented Apr 12, 2023 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.