# Hierarchical Clustering: Extract observations from large heatmap

I'm currently trying to visualize a large data set as heat map. That in itself works smoothly but I struggle with gaining insights from interestingly looking clusters.

Specifically, I have two questions that are very related:

First, I find clusters of interesting features and am looking for a systematic way to extract the a flat clustering at a specific level (but the fcluster function seems to do something different and cut_tree doesn't work with those trees). I would like to have a slice of the hierarchical clustering at a specified depth of the dendrogram. This is probably encoded in the linkage matrix Z but I struggle to understand how exactly I can extract that information from Z.

Second, with the complicated heatmap pictured below, the row names on the right are gene names for every 100th data point (gene). I would now like to have a look at which genes are in some of the little clusters, for instance the little black square for feature MF: LIHC that is marked. I know the IDs of the genes that are labelled on the right, so I would want to know something like:

Which genes are in the same cluster as CAPN7 at level 5?

Roman

A similar question was asked at stackoverflow. There it was proposed to use the option criterion='maxclust' for the flcuster function:

from scipy.cluster.hierarchy import fcluster
clust = fcluster(Z, t=k, criterion='maxclust')


The description of this option in the flcuster documentation is a bit confusing, but that's how you get a clustering with t=k clusters.

You should be able to retrieve the answer to your second question with the resulting array.

I'm not sure what you are referring to in Q1, but for Q2, it looks like you are trying to dig down to the lower negative correlated items, right. I can't reproduce your exace example (I don't have the data), but I will give you a generic example, which you can easily adapt to your specific scenario.

# get only numerics from your dataframe; correlations work on values not labels
df = df.sample(frac=0.1, replace=True, random_state=1)
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
newdf = df.select_dtypes(include=numerics)

for col in newdf.columns:
print(col)

# Compute the correlation matrix
# no statistically significant correlations between any numeric features...
corrmat = newdf.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(15,15))
#plot heat map
g=sns.heatmap(newdf[top_corr_features].corr(),annot=True,cmap="RdYlGn")


# Identify Highly Negatively Correlated Features
# Create correlation matrix
corr_matrix = newdf.corr()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
# Find index of feature columns with correlation less than 0
to_keep = [column for column in upper.columns if any(upper[column] < 0)]


Answer: ['rating', 'num_comments', 'list_price', 'lowest_price_new_condition']

Finally, let's say you want to drop everything that has >.2 correlation, and keep only the non-correlated features or negatively correlated features, you can do this...

# Create correlation matrix
corr_matrix = newdf.corr()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
# Find index of feature columns with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > .2)]
# Drop features
finaldf = newdf.drop(newdf[to_drop], axis=1)

corrmat = finaldf.corr()
top_corr_features = corrmat.index
plt.figure(figsize=(15,15))
#plot heat map
g=sns.heatmap(finaldf[top_corr_features].corr(),annot=True,cmap="RdYlGn")