# Cramer von mises test requires too much resources

I am doing some data analysis, and I am using the cramer von mises test to check if two samples are derived from the same distribution. I am using the cramervonmises_2samp implementation from scipy.stats. Originally, the samples that I was using (denoted sample1 and sample2) were 800K and 200K observations. When I tried to compute this, it took quite a bit, and it ate up all my ram (32GB), causing my jupyter notebook kernel to die.

My statistical test function as defined as:

def compare_distributions(sample1, sample2, ALPHA=.01):

res = cramervonmises_2samp(sample1, sample2, method='exact')

print(f'sample1 shape: {sample1.shape}')
print(f'sample2 shape: {sample2.shape}')

print(f'statistic: {res.statistic}')
print(f'p-value: {res.pvalue}')

if res.pvalue < ALPHA:
print('The null hypothesis is rejected. \nThe samples do not have the same distribution.\n\n')
else:
print('The null hypothesis is not rejected. \nThe samples are drawn from the same distribution.\n\n')



As an experiment to see how much data my machine can handle I decided to do the test with 50, 70, 100, 130 and 200 observations in each sample, with the samples being from scipy.stats.gamma and scipy.stats.expon respectively. The results are as follows:

sample1 shape: (50,)
sample2 shape: (50,)
statistic: 1.8978000000000002
p-value: 1.5142491077016167e-05
The null hypothesis is rejected.
The samples do not have the same distribution.

Runtime: 6.093s

----------------------------------------------

sample1 shape: (70,)
sample2 shape: (70,)
statistic: 2.2905102040816345
p-value: 1.9285375980904996e-06
The null hypothesis is rejected.
The samples do not have the same distribution.

Runtime: 33.788s

----------------------------------------------

sample1 shape: (100,)
sample2 shape: (100,)
statistic: 3.0496999999999943
p-value: 3.495231315392338e-08
The null hypothesis is rejected.
The samples do not have the same distribution.

Runtime: 199.694s

----------------------------------------------

sample1 shape: (130,)
sample2 shape: (130,)
statistic: 5.4970118343195296
p-value: 5.803075867500237e-14
The null hypothesis is rejected.
The samples do not have the same distribution.

Runtime: 783.185s



When I tried to do the test with 200 observations in each sample, this time, my laptop crashed and was in need of a restart.

My questions are:

• why does this test require so much resources? (in terms of ram and runtime)
• is hypothesis testing always so resource intensive?

This is because of the fact you are trying to calculate the exact p-value by using method='exact'. Try using method='asymptotic', as that should speed up the calculations quite a bit (2ms vs 9.2s for 50 samples on my computer) with what I suspect little to no difference in the calculated p-value.
• well yes, but, how do I know how close is the asymptotic method to the exact one? And judging by the cumulative distribution function plot on some of the other samples, they look nearly identical, yet the asymptotic calculation rejects the null hypothesis. @Oxbowerce do you know how method='asymtotic' actually calculates the result? Commented Apr 12, 2023 at 18:52