Hierarchical Clustering: Extract observations from large heatmap

Question

I'm currently trying to visualize a large data set as heat map. That in itself works smoothly but I struggle with gaining insights from interestingly looking clusters.

Specifically, I have two questions that are very related:

First, I find clusters of interesting features and am looking for a systematic way to extract the a flat clustering at a specific level (but the fcluster function seems to do something different and cut_tree doesn't work with those trees). I would like to have a slice of the hierarchical clustering at a specified depth of the dendrogram. This is probably encoded in the linkage matrix Z but I struggle to understand how exactly I can extract that information from Z.

Second, with the complicated heatmap pictured below, the row names on the right are gene names for every 100th data point (gene). I would now like to have a look at which genes are in some of the little clusters, for instance the little black square for feature MF: LIHC that is marked. I know the IDs of the genes that are labelled on the right, so I would want to know something like:

Which genes are in the same cluster as CAPN7 at level 5?

Thank you for your help!

Roman

Answer 1

A similar question was asked at stackoverflow. There it was proposed to use the option criterion='maxclust' for the flcuster function:

from scipy.cluster.hierarchy import fcluster
clust = fcluster(Z, t=k, criterion='maxclust')

The description of this option in the flcuster documentation is a bit confusing, but that's how you get a clustering with t=k clusters.

You should be able to retrieve the answer to your second question with the resulting array.

Answer 2

I'm not sure what you are referring to in Q1, but for Q2, it looks like you are trying to dig down to the lower negative correlated items, right. I can't reproduce your exace example (I don't have the data), but I will give you a generic example, which you can easily adapt to your specific scenario.

# get only numerics from your dataframe; correlations work on values not labels
df = df.sample(frac=0.1, replace=True, random_state=1)
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
newdf = df.select_dtypes(include=numerics)

for col in newdf.columns: 
    print(col) 

# Compute the correlation matrix
# no statistically significant correlations between any numeric features...
corrmat = newdf.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(15,15))
#plot heat map
g=sns.heatmap(newdf[top_corr_features].corr(),annot=True,cmap="RdYlGn")

# Identify Highly Negatively Correlated Features
# Create correlation matrix
corr_matrix = newdf.corr()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
# Find index of feature columns with correlation less than 0
to_keep = [column for column in upper.columns if any(upper[column] < 0)]

Answer: ['rating', 'num_comments', 'list_price', 'lowest_price_new_condition']

Finally, let's say you want to drop everything that has >.2 correlation, and keep only the non-correlated features or negatively correlated features, you can do this...

# Create correlation matrix
corr_matrix = newdf.corr()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
# Find index of feature columns with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > .2)]
# Drop features 
finaldf = newdf.drop(newdf[to_drop], axis=1)

corrmat = finaldf.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(15,15))
#plot heat map
g=sns.heatmap(finaldf[top_corr_features].corr(),annot=True,cmap="RdYlGn")