I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW) is a useful method to find alignments between two time series which may vary in time or speed.

I have found dtw_std in mlpy library and scipy.cluster.hierarchy in SciPy in order to cluster my data.

From the scipy docs, I find that I could use my custom distance function:

metric : str or function, optional The distance metric to use in the case that y is a collection of observation vectors; ignored otherwise. See the pdist function for a list of valid distance metrics. A custom distance function can also be used.

But I am stuck matching this information to implement clustering.

My dataset is in the format of dataframe which each row corresponds to a sample.

Here is my questions:

1- How can I provide distance matrics for the linkage function?

2- How to set my custom distance function?

import pandas as pd
import scipy.cluster.hierarchy as hac
import mlpy

dataset = pd.read_csv ( "dataset.csv",encoding='utf-8' )
X # distance matrics
Z = hac.linkage(X, metrics=mlpy.dtw_std, method='average')
cluster = hac.fcluster(Z, t=100, criterion='maxclust')
leader = hac.leaders(Z, t=100, criterion='maxclust')

fig = plt.figure(figsize=(25, 10))
dn = dendrogram(Z)


Here is how I compute distance matrix, then I pass it to linkage:

# computing distance matrix
dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))
z = hac.linkage(dm, method='average')
cluster = hac.fcluster(z, t=100, criterion='maxclust')
leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='maxclust')

Use a precomputed distance matrix, and distance="precomputed".

HAC will compute a distance matrix anyway.

  • $\begingroup$ I have added the way I compute distance matrix. Where I should set distance="precomputed"? which function in HAC will compute distance matrix anyway? $\endgroup$ Dec 27 '17 at 12:47
  • $\begingroup$ Sorry, I was having the sklearn not the SciPy API in mind. IIRC, SciPy always needs a special serialized upper triangular matrix. You can find a conversion function from a square distance matrix to the desired format. See the docs. $\endgroup$ Dec 27 '17 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.