I know what mutual information basically is but not quite sure about why and how it is used in the context of evaluation of clustering mechanisms ? Can someone please explain the intuition behind it ? i.e, how it is defined in the case of clustering evaluation ?


1 Answer 1


This is a method for evaluation of two clusterings in the presence of class labels so it is not proper for real clustering problems in which class labels are not available.

Imagine you have class labels and you want to evaluate a clustering or (compare two clusterings). The most natural idea is to use Purity score. It simply checks labels with clusters and the best case is, of course, when each cluster contains only and only one class label. This score, however seemingly natural, has a drawback. If you consider each cluster having only one data point, then Purity is maximized! So there should be an awareness about the number of clusters when calculating the purity score.

The next idea is calculating the Mutual Information. Mutual Information considers two splits: (1) split according to clusters and (2) split according to class labels. Then it tells you how these two splittings agree each other (how much information they share about each other or how can you know about one of them if you know the other one). Like purity, MI also gets bigger when the number of clusters is large.

Then comes NMI which is bias-corrected for the phenomenon explained above and also normalizes the score between $0$ and $1$ (MI does not have an upper bound).

NOTE: I think your question was answered in first line. If you want to evaluate clustering you are not looking for external measures where labels are needed. I just explained a bit for sake of completeness of answer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.