5
$\begingroup$

Context

I am involved in the task of clustering 1500 time series of 500 observations into a few clusters. The time series share all the same observed properties at different spatial locations, but responding to the same exogenous variables. However, for each time series, the magnitude of the response is very different. For a time series of reference $X$, I would like to be grouped in the same cluster series that are alike $X^a$ for all $a > 0$.

Tryouts

So far, my interpretation of the problem is that I want to cluster time series sharing a strong monotonic relationship. My first tryouts used hierarchical agglomerative clustering by defining a distance-based Kendall's tau rank coefficient since it measures the strength of a monotonic relationship. By visual interpretation, the best results were obtained using Ward's linkage method. However, this approach seems unorthodox, non-robust, or doubtful for several reasons.

First, Scipy documentation mentions here that Ward's method is only correct when the Euclidean distance is used. Secondly, I couldn't find any detailed application of time series clustering based either on Spearman or Kendall's tau coefficient. Furthermore, I was very surprised that I couldn't find any paper or reference aiming at clustering based on a monotonic criterion.

I am willing to consider other approaches, though I cannot measure their benefits. For instance, rescaling all time-series to map them to a standardized Gaussian distribution (e.g. Box-Cox) and then using the Euclidean distance. Another possibility is to turn the first difference of time series into a boolean vector (1 if $\Delta X >0$, $0$ otherwise) and then use the Euclidean distance or another distance metric.

Questions

Since I am new to time series clustering, I have some trouble picturing by myself what would be the best approach(es) (or the worse) for this specific purpose. Hence, I have two related questions:

  1. Specifically, is using Hierarchical Clustering based on Kendall's tau and Ward's linkage method a wrong way to go, and why?
  2. Generally, what is the best way to cluster time series based on monotonic association?

Some references on the topic are also welcome.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.

Why don't you consider average linkage? Why Ward?

Improve this answer
$\endgroup$
2
  • $\begingroup$ As it is a spatiotemporal dataset, I am able to check how clusters are spatially distributed. Kendall + Average linkage yields to one dominant cluster while the others are small and spatially split. Ward + Kendall yield to more consistent clusters with respect to the spatial structure of the system. However, I am able to get a similar spatial distribution using Pearson's correlation on Z standardized data with Ward linkage. This is even better if I apply a quantile mapping to a normal distribution on my time series. I guess it might be more be "mathematical" than relying on Kendall's tau. $\endgroup$
    – Delforge
    Commented Feb 19, 2019 at 10:26
  • $\begingroup$ Pearson does its own normalization... And it's easy to prove that it is related to Euclidean on appropriately preprocessed data. So if you do this math, you may be able to set it up so that you can use Euclidean (and hence Ward) to get Pearson results. You just need to show that you can pull out the normalization, and that it won't yield a division by 0 on your data. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 19, 2019 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.