1
$\begingroup$

I'm hoping to get some answers to a question I have regarding normalization of DTW datasets, in particular datasets in which two time-series shapes with similar shapes but differences in magnitude are misclassified under z-normalization.

One reason is simply empirical. When I z-normalize my data it doesn't cluster well. When I don't z-normalize it works quite well. But that isn't really enough to ignore the normalization advice.

The second is in reading the links in this thread which does a good job summarizing the other threads related to this, reading the literature, and thinking about my data. I cannot find a good discussion of the case where magnitude matters within a subset of similar clusters and z-normalization negates that important magnitude difference. In cases where the shape of some data can generally be the same (the lengths are the same, but the amplitude differs) z-normalization makes them cluster as identical when they aren't. Does this mean that DTW just isn't the right tool, that I should be normalizing differently, or is my data a special case because it is already normalized the way it is collected?

Briefly, my data is isotopic ratios recorded in fish "ear stones" which change as a fish moves from one location to another and are recorded in the growing layers of the stone (called an otolith). The isotope ratio in each river is very stable, and that ratio is known to the third or fourth decimal place. There is no biological fractionation, the information is recorded as it exists in the river when the fish enters the river. All data is normalized at collection with regard to the global ocean value which is known to the 5th decimal place. What we care about is classifying the data according to the shape of the isotopic curve, which is characteristic of where and when fish moved between rivers, the different decisions fish make in their movements. All the curves are interpolated to the same length prior to analysis.

The problem I see in my data is illustrated by the conceptual plot below.

enter image description here

The warping of DTW helps to better classify fish with mild differences in their movement timing, like the two black lines in the plot. It also does a good job of classifying patterns with really different shapes (the red line). Without z-normalization, DTW correctly classifies the blue line as a different group. But, using z-normalization, the two black lines classify with the blue line because the magnitude difference is erased.

To take the example from Section 1.2.1 of Searching and Mining Trillions of Time Series Subsequences under Dynamic Time Warping from the linked question above, the movement of fish in two different habitats is analogous to the action of drawing a gun or pointing a finger. The reason for z-normalization in the gun-point dataset, according to the author, is to account for offsets in scaling of the video. But in this case all of the data is normalized already to the global marine signature to account for machine drift, and the isotopic values are extremely precise and accurate. The error in amplitude of the curves is less than 1%. Is z-normalization required here, or is it just erasing meaningful differences in amplitude that are correctly classified without z-normalization?

I'm curious not just to find an answer regarding my own data, but also because it has been really difficult for me to find a straightforward discussion of this type of amplitude problem in the literature. Most of the advice is a blanket, "In 99% of cases, you must z-normalize." as eamonn stated in the other question...an opinion I respect given his work on this subject. But, I'm curious about what those 1% of examples are. Perhaps I'm just missing it, or perhaps I'm missing the boat completely. Your thoughts, and explanations, would be greatly appreciated.

$\endgroup$
0
$\begingroup$

That eamonn guys sounds smart.

If you do not normalize, small differecnes in the mean and/or STD rapidly swamp any shape similarity.

Here is one way to think about it.

If you do not normalize, and the means or STDs are different, then.......

EuclideanDIST(A,B) is approximately equal to DTWdist(A,B), which is approximately equal to [abs(mean(A)-mean(B)) times a constant]

If you can model the similairy with just the mean, they why do you need to even look at the shape?

BTW, you have a cool problem/dataset

| improve this answer | |
$\endgroup$
  • $\begingroup$ "If you can model the similairy with just the mean, they why do you need to even look at the shape?" Good question. We could, and do. But there is a lot going on between each of those stable areas in the life of a fish that is hard to easily quantify. The plot is a big simplication of the variation in movement. DTW seems like a good way to integrate all of that movement, since a given life-history would result in a given shape. But, if it requires that all magnitude information be removed, then certain life histories look identical in terms of shape...but are different, which is a problem. $\endgroup$ – JHegg Apr 28 at 5:56
  • $\begingroup$ So...as a solution, I could split the data between series with high standard deviation and low standard deviation based on some criteria I know to be true within my data (we know the means of the rivers and what movements are possible after all), then z-norm and cluster each group? That would avoid individuals with similar shapes but different SD being clustered together. Or separate clusters after the fact based on a similar criteria. Or, is there a time series clustering method that takes magnitude into account as well? $\endgroup$ – JHegg Apr 28 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.