Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have some observations belonging to groups and I would like to compute the similarity of them within different groups in order to tell which observations, within specific groups, have similar characteristics or not.
Which metrics can be useful to do such things and under which conditions?
I know that there are many metrics to compute the similarities between individuals in a group, but there is no threshold to tell whether individuals in a cluster share similar patterns or not.
I would like to know which metrics can be useful to do such things
Since you are clustering your data, why not use the same metric that you used during clustering for comparing your data inside your clusters? I assume this should technically give you exactly what you need.
there is no threshold to tell whether individuals in a cluster share similar pattern or not
This hardly depends on your data and what you want to do with your similarity. If you just want to retrieve the most similar point, you can do it pretty straight forward.
If you, on the other hand, want to remove outliers using this measure, you can surely also do so, but you might as well just use an algorithm that deals with outliers in the first place.
This seems to me like a hypothesis testing problem where the null hypothesis can be formulated as "two data sets are drawn from the same population distribution function". If we can disprove this statement with sufficient certainty then we can assume that they were in fact drawn from different populations.
You can bin your features into ranges and then apply the Chi-Squared Test defined as
$\chi^2 = \sum_i^n \frac{(O_i-E_i)^2}{E_i}$
where $i$ is each bin you have for a feature, $O_i$ is the number of observations you have for the $i^{th}$ bin, $E_i$ is the expected number of observations. $E_i = Np_i$, where $N$ is your number of observations and $p_i$ is the probability of obtaining bin $i$.
For example assume a simple coin flip experiment where we want to determine if a coin is fair. The baseline distribution should be uniform 50 heads, 50 tails. We will conduct a chi-squared test to see if our coin is fair. When flipping the coin we observed 30 heads and 70 tails.
$\chi^2 = \frac{(30 - 100*0.5)^2}{100*0.5} + \frac{(70-100*0.5)^2}{100*0.5} = 16$
This example has 1 degree of freedom. There are two possible outcomes, $r = (n-1)$.
Now we can look at our trusty chi-squared table. The rows is usually for the degrees of freedom $\nu$, which is 1 in our example. And the columns is for the level of significance, for a significant result look under $p<0.05$ for a highly significant result look under $p<0.01$. We can see that our calculated $\chi^2$ result is much higher than the value for a $p<0.01$ thus we can say witth certainty that the distribution of our Group 2 is not the same as that of Group 1. They are statistically different.
Alternatively, you can get the probability distribution function for each variable and get the overlapping area between two groups.
