1
$\begingroup$

I am comparing with Chi Square the distributions of two categorical variables. Both have the same number of classes. After counting each class per variable, I obtain very similar counts but the p-value result of the chi-square test is 0 - rejecting the null hypothesis. I am not sure what I am missing.

Here is the code:

    import numpy as np
from scipy.stats import chi2_contingency

var1_arr = np.array([361837, 94360, 1533308]) # counts per class for var 1
var2_arr = np.array([355572, 93285, 1544745]) # counts per class for var 2

observed_counts = np.vstack((var1_arr,var2_arr))

# # Given class counts
# observed_counts = np.array([[361837, 94360.67, 1533308.67],
#                             [355572, 93285, 1544745]])

# Calculate expected frequencies
N = observed_counts.sum()
expected_counts = (observed_counts / N) * N

# Perform chi-square test
chi2, p_value, dof, expected = chi2_contingency(observed_counts)

print(f"Chi-Square Statistic: {chi2:.4f}")
print(f"P-value: {p_value:.4f}")

The result is: Chi-Square Statistic: 99.1516 P-value: 0.0000

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You have a huge number of observations. Consequently, the test is highly sensitive to small differences. The test has considerable statistical power to detect these.

You are within your rights to say that these differences lack practical significance, but a tiny p-value is not surprising.

Significance test for large sample sizes

Improve this answer
$\endgroup$
4
  • $\begingroup$ Indeed, I start to remember now. If I am not wrong it I couldn't find neither in the past a feasible solution to large samples $\endgroup$
    – crbl
    Commented Apr 16 at 14:59
  • $\begingroup$ @crbl What do you mean by a "solution" to large samples? $\endgroup$
    – Dave
    Commented Apr 16 at 15:06
  • $\begingroup$ I am referring to a test with less 'false positives' which will highlight only important differences between the two distributions, even for large samples. $\endgroup$
    – crbl
    Commented Apr 17 at 12:58
  • $\begingroup$ @crbl Then you’re not looking for hypothesis testing, which only highlights likely differences (loosely speaking), not important differences. Please refer to the link in my answer where I liken this to the Princess and the Pea fairy tale. $\endgroup$
    – Dave
    Commented Apr 17 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.