I have two data sets defined by real valued vectors, and I have performed clustering on both of them. Now I want to compare the classes to see how they map to each other. If I put the data sets together into a single, bigger data set, and cluster that, I see that most of, say, class 1 of the first data set and class 2 of the second (numbers are arbitrary) fall into the same class of the total clustering, which makes sense, meaning they're probably equivalent. What would be a good rigorous way of performing this kind of analysis and giving a quantitative measure of this overlap? Thanks.
I suggest using Kullback–Leibler divergence (KLD) to compare the classes. The procedure is as follows:
1- Find the mean vector and covariance matrix of each class of each dataset.
2- Calculate the KLD between each class of the first dataset and all classes of the second dataset. So, for example if you have two dataset and each of them has 5 classes, you should calculate KLD, 25 times.
3- For each class of the first dataset, find the class from the second dataset with the lowest KLD between them. Doing this, you will find the most similar classes of the two datasets.
Note that KLD measures the similarity between two probability distributions and by calculating the mean vector and covariance matrix of each class, you are fitting a normal distribution to it. To find the formula for calculation of KLD between two multivariate normal distribution, refer to here:
The usual cluster evaluation measures can be used, and this will be easiest to convey to reviewers and clustering experts.
You simply restrict the full labels to only one part of the data, then compare established measures such as the adjusted Rand index (ARI) or normalized mutual information (NMI) on the two results (these measures are symmetric, so it does not matter which labels are from the full data clustering, and which are from the subset clustering.