Correlation between categorical variables based on the target distribution

Question

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way to combine them (lets say that $X_1$ and $X_3$ are very similar in the frequency of the there $Y$ distribution) where $X$ in ($x_1,...,x_n$) and y in ($y_1,...,y_n$)

What is the best way to find all this $X$ subgroups that have similar $Y$ distribution? the reason I'm doing this is because I know that lots of my $X$s are the same but someone labeled them as different.

I have started with doing spearman corr matrix over the frequency table for each $X$ and its $Y$s distribution , but I'm not sure it is the right way to handle it and it gives me some bad results

Answer 1

Your answer is $\chi^2$ (chi-squared) test of independence.

First we have to calculate the expected value of the two nominal variables. We can calculate the expected value of the two nominal variables by using this formula:

$E_{i,j} = \frac{\sum_{k=1}^cO_{ik}*\sum_{k=1}^rO_{kj}}{N}$

Where,

$E_{ij}$ = Expected value of the cell $i$, $j$

$\sum_{j=1}^cO_{ij}$ = Sum of the ith column

$\sum_{k=1}^rO_{kj}$ = Sum of the kth row

$N$ = total number

After calculating the expected value, we will apply the following formula to calculate the value of the Chi-Square test of Independence:

$\chi^2 = \sum\sum\frac{(O_{ij}-E_{ij})^2}{E_{ij}} $: Chi-Square test of Independence

$O_{ij}$ = Observed value of two nominal variables

$E_{ij}$ = Expected value of two nominal variables

Degree of freedom is calculated by using the following formula: $df = (r-1)(c-1)$

Where, DF = Degrees of freedom r = number of rows c = number of columns

Hypothesis:

Null hypothesis: Assumes that there is no association between the two variables.

Alternative hypothesis: Assumes that there is an association between the two variables.

Hypothesis testing: Hypothesis testing for the chi-square test of independence as it is for other tests like ANOVA, where a test statistic is computed and compared to a critical value. The critical value for the chi-square statistic is determined by the level of significance (typically .05) and the degrees of freedom. The degrees of freedom for the chi-square are calculated using the following formula: df = (r-1)(c-1) where r is the number of rows and c is the number of columns. If the observed chi-square test statistic is greater than the critical value, the null hypothesis can be rejected.

Answer 2

Adding to two good answers, here is a non statistical testing option: Bhattacharyya distance

Answer 3

Leveraging Juan's answer you can also use the Kolmogorov-Smirnov-Test to test wether two variables are drawn from the same distribution.

As outlined by Bruce Mitchell (1971) - "A Comparison of Chi-Square and Kolmogorov-Smirnov Tests", the KS test can be used instead of chi²-test, if not all requirements are satisfied and tends to be more flexible in its application.