hierarchical clustering doesn't work as expected

Question

I have a precomputed distance matrix.

I'm trying to do an hierarchical clustering using scipy:

from scipy.cluster.hierarchy import fcluster, linkage, cut_tree,fcluster

z = linkage(ds, "average")
clust = fcluster(z, 4, criterion='maxclust')
print(clust)

The problem is that, most of the labels ends up in one group - Why?

By the way, I tried to use AgglomerativeClustering as well but I got similar results.

ds is the precomputed distance matrix. I'm using DTW, a distance metric for time series analysis.

I uploaded ds.csv to here.

Also, here's an heatmap which forms interesting lines (I don't really know how to interpret this currently)

Answer 1

Likely your precomputed distance matrix is bad.

Judging from your plot, it's probably a half-filled similarity matrix instead. And it likely is not in the required form for linkage.

The input y may be either a 1d condensed distance matrix or a 2d array of observation vectors.

Did you make your distance matrix "condensed"? Otherwise it will be treated as data matrix, not distance matrix, because python does not have type safety but uses duck typing - if it quacks like a data matrix, it is a data matrix. Then you end up computing Euclidean distances on half-filled rows of this matrix...