Hierarchical clustering with the consensus matrix as similarity matrix

Question

I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help assess the clusters’ composition andnumber. In particular, if we associate a color gradient to the 0–1 range of real numbers, so that white corresponds to 0, and dark red corresponds to 1, and if we assume the matrix is arranged so that items belonging to the same cluster are adjacent to each other (with the same item order used to index both the rows and the columns of the matrix), a matrix corresponding to perfect consensus will be displayed as a color-coded heat map characterized by red blocks along the diagonal, on a white background.", see the image below.

The consensus matrix itself is a (N × N) matrix that stores, for each pair of items, the proportion of clustering runs in which two items are clustered together. The consensus matrix is obtained by taking the average over the connectivity matrices of every perturbed dataset. To go from a consensus matrix to the visualization above, the authors say the following: "We can use the consensus matrix itself to determine the optimal item order. In particular, if we carry out hierarchical clustering with the consensus matrix as similarity matrix, the induced dendogram will have its leaves arranged so as to have items with highest consensus index adjacent to each other, thus maximizing the block-diagonal nature of the heat map ordered accordingly (it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix)."

I have my consensus matrix after 10 times sampling and clustering my dataset with 500 items. The consensus matrix looks like:

Where a value of 1 or 0 indicates perfect agreement/consensus on the clustering results of the 10 times sampling. However, I fail to understand how we end up with the visualization above. I have a few questions:

1. The authors state that we use the consensus matrix as a similarity matrix, but doesn't hierarchical clustering need a distance matrix? For example in the Python sklearn implementation of AgglomerativeClustering it says: "If “precomputed”, a distance matrix (instead of a similarity matrix) is needed as input for the fit method."
2. What do the authors mean with "it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix"?

Answer 1

To address your two questions:

1. Agglomerative clustering requires a distance metric, but you can compute this from your consensus-similarity matrix. The most basic way, is to do this:

   distance_matrix = 1 / similarity matrix

   Although, they may explicitly state in the paper what function they use for this transformation.

2. I think this is just to say that the matrix is symmetric. The x-axis of the heatmap will be n=0,1,2,3,4 and the y-axis of the heatmap will be n=0,1,2,3,4. This is the same procedure as for a correlation matrix. Just keep your matrix as is, and it will keep that order.