1
$\begingroup$

Let $X$ be a category with very high cardinality and $Y$ be my target. when I look at $X$ distribution to $Y$ I see that some of the levels are very similar to each other . I would like to find a way to combine them (lets say that $X_1$ and $X_3$ are very similar in the frequency of the there $Y$ distribution) where $X$ in ($x_1,...,x_n$) and y in ($y_1,...,y_n$) enter image description here

What is the best way to find all this $X$ subgroups that have similar $Y$ distribution? the reason I'm doing this is because I know that lots of my $X$s are the same but someone labeled them as different.

I have started with doing spearman corr matrix over the frequency table for each $X$ and its $Y$s distribution , but I'm not sure it is the right way to handle it and it gives me some bad results

$\endgroup$

3 Answers 3

2
$\begingroup$

Your answer is $\chi^2$ (chi-squared) test of independence.

First we have to calculate the expected value of the two nominal variables. We can calculate the expected value of the two nominal variables by using this formula:

$E_{i,j} = \frac{\sum_{k=1}^cO_{ik}*\sum_{k=1}^rO_{kj}}{N}$

Where,

$E_{ij}$ = Expected value of the cell $i$, $j$

$\sum_{j=1}^cO_{ij}$ = Sum of the ith column

$\sum_{k=1}^rO_{kj}$ = Sum of the kth row

$N$ = total number

After calculating the expected value, we will apply the following formula to calculate the value of the Chi-Square test of Independence:

$\chi^2 = \sum\sum\frac{(O_{ij}-E_{ij})^2}{E_{ij}} $: Chi-Square test of Independence

$O_{ij}$ = Observed value of two nominal variables

$E_{ij}$ = Expected value of two nominal variables

Degree of freedom is calculated by using the following formula: $df = (r-1)(c-1)$

Where, DF = Degrees of freedom r = number of rows c = number of columns

Hypothesis:

Null hypothesis: Assumes that there is no association between the two variables.

Alternative hypothesis: Assumes that there is an association between the two variables.

Hypothesis testing: Hypothesis testing for the chi-square test of independence as it is for other tests like ANOVA, where a test statistic is computed and compared to a critical value. The critical value for the chi-square statistic is determined by the level of significance (typically .05) and the degrees of freedom. The degrees of freedom for the chi-square are calculated using the following formula: df = (r-1)(c-1) where r is the number of rows and c is the number of columns. If the observed chi-square test statistic is greater than the critical value, the null hypothesis can be rejected.

$\endgroup$
1
  • $\begingroup$ Some of the Y's might be zero for both Xi and Xj , so running chi-squared test of independence ended in dividing by zero .. i think about it as two count vectors that i would like to know how similar are they in the pattern they hold.. $\endgroup$
    – Latent
    Jan 6, 2020 at 8:50
2
$\begingroup$

Adding to two good answers, here is a non statistical testing option: Bhattacharyya distance

$\endgroup$
1
$\begingroup$

Leveraging Juan's answer you can also use the Kolmogorov-Smirnov-Test to test wether two variables are drawn from the same distribution.

As outlined by Bruce Mitchell (1971) - "A Comparison of Chi-Square and Kolmogorov-Smirnov Tests", the KS test can be used instead of chi²-test, if not all requirements are satisfied and tends to be more flexible in its application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.