1
$\begingroup$

I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help assess the clusters’ composition andnumber. In particular, if we associate a color gradient to the 0–1 range of real numbers, so that white corresponds to 0, and dark red corresponds to 1, and if we assume the matrix is arranged so that items belonging to the same cluster are adjacent to each other (with the same item order used to index both the rows and the columns of the matrix), a matrix corresponding to perfect consensus will be displayed as a color-coded heat map characterized by red blocks along the diagonal, on a white background.", see the image below.

enter image description here

The consensus matrix itself is a (N × N) matrix that stores, for each pair of items, the proportion of clustering runs in which two items are clustered together. The consensus matrix is obtained by taking the average over the connectivity matrices of every perturbed dataset. To go from a consensus matrix to the visualization above, the authors say the following: "We can use the consensus matrix itself to determine the optimal item order. In particular, if we carry out hierarchical clustering with the consensus matrix as similarity matrix, the induced dendogram will have its leaves arranged so as to have items with highest consensus index adjacent to each other, thus maximizing the block-diagonal nature of the heat map ordered accordingly (it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix)."

I have my consensus matrix after 10 times sampling and clustering my dataset with 500 items. The consensus matrix looks like:

enter image description here

Where a value of 1 or 0 indicates perfect agreement/consensus on the clustering results of the 10 times sampling. However, I fail to understand how we end up with the visualization above. I have a few questions:

  1. The authors state that we use the consensus matrix as a similarity matrix, but doesn't hierarchical clustering need a distance matrix? For example in the Python sklearn implementation of AgglomerativeClustering it says: "If “precomputed”, a distance matrix (instead of a similarity matrix) is needed as input for the fit method."
  2. What do the authors mean with "it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix"?
$\endgroup$

1 Answer 1

1
$\begingroup$

To address your two questions:

  1. Agglomerative clustering requires a distance metric, but you can compute this from your consensus-similarity matrix. The most basic way, is to do this:

    distance_matrix = 1 / similarity matrix

    Although, they may explicitly state in the paper what function they use for this transformation.

  2. I think this is just to say that the matrix is symmetric. The x-axis of the heatmap will be n=0,1,2,3,4 and the y-axis of the heatmap will be n=0,1,2,3,4. This is the same procedure as for a correlation matrix. Just keep your matrix as is, and it will keep that order.

$\endgroup$
1
  • $\begingroup$ Let me know if you have any questions otherwise please accept this answer $\endgroup$
    – WBM
    Mar 4, 2021 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.