27
$\begingroup$

Background

I'm working on a time series data set of energy meter readings. The length of the series varies by meter - for some I have several years, others only a few months, etc. Many display significant seasonality, and often multiple layers - within the day, week, or year.

One of the things I've been working on is clustering of these time series. My work is academic for the moment, and while I'm doing other analysis of the data as well, I have a specific goal to carry out some clustering.

I did some initial work where I calculated various features (percentage used on weekends vs. weekday, percentage used in different time blocks, etc.). I then moved on to looking at using Dynamic Time Warping (DTW) to obtain the distance between different series, and clustering based on the difference values, and I've found several papers related to this.

Question

Will the seasonality in a specific series changing cause my clustering to be incorrect? And if so, how do I deal with it?

My concern is that the distances obtained by DTW could be misleading in the cases where the pattern in a time series has changed. This could lead to incorrect clustering.

In case the above is unclear, consider these examples:

Example 1

A meter has low readings from midnight until 8AM, the readings then increase sharply for the next hour and stay high from 9AM until 5PM, then decrease sharply over the next hour and then stay low from 6PM until midnight. The meter continues this pattern consistently every day for several months, but then changes to a pattern where readings simply stay at a consistent level throughout the day.

Example 2

A meter shows approximately the same amount of energy being consumed each month. After several years, it changes to a pattern where energy usage is higher during the summer months before returning to the usual amount.

Possible Directions

  • I've wondered whether I can continue to compare whole time series, but split them and consider them as a separate series if the pattern changes considerably. However, to do this I'd need to be able to detect such changes. Also, I just don't know if this is a suitable way or working with the data.
  • I've also considered splitting the data and considering it as many separate time series. For instance, I could consider every day/meter combination as a separate series. However, I'd then need to do similarly if I wanted to consider the weekly/monthly/yearly patterns. I think this would work, but it's potentially quite onerous and I'd hate to go down this path if there's a better way that I'm missing.

Further Notes

These are things that have come up in comments, or things I've thought of due to comments, which might be relevant. I'm putting them here so people don't have to read through everything to get relevant information.

  • I'm working in Python, but have rpy for those places where R is more suitable. I'm not necessarily looking for a Python answer though - if someone has a practical answer of what should be done I'm happy to figure out implementation details myself.
  • I have a lot of working "rough draft" code - I've done some DTW runs, I've done a couple of different types of clustering, etc. I think I largely understand the direction I'm taking, and what I'm really looking for is related to how I process my data before finding distances, running clustering, etc. Given this, I suspect the answer would be the same whether the distances between series are calculated via DTW or a simpler Euclidean Distance (ED).
  • I have found these papers especially informative on time series and DTW and they may be helpful if some background is needed to the topic area: http://www.cs.ucr.edu/~eamonn/selected_publications.htm
Improve this question
$\endgroup$
5
  • $\begingroup$ +1 Very nice question, and it is great to see so much enthusiasm! I think you could nail down your question a little bit, so it's more inviting for others to read, and then give you an answer. $\endgroup$
    – Rubens
    Dec 22, 2014 at 9:56
  • $\begingroup$ @Rubens Thanks! I'll re-work it when I'm home this evening, I can see where it'd be useful to include some more information about how I've gotten to this point and why. I was worried about it getting too long, but I'll separate out the background and question a bit more to avoid it getting unreadable. $\endgroup$
    – Jo Douglass
    Dec 22, 2014 at 10:04
  • $\begingroup$ It may not be a "pure statistics" question but it needs a pure statistics answer. You will struggle until you can think about it in pure statistics terms. $\endgroup$
    – Spacedman
    Dec 26, 2014 at 11:02
  • $\begingroup$ @Spacedman - I welcome answers in whatever manner people feel is the best way to answer it, with the caveat that I may have further questions if the answer is heavy on formulas or references to statistical concepts that I don't understand yet. $\endgroup$
    – Jo Douglass
    Dec 27, 2014 at 13:55
  • $\begingroup$ Jo did you find the right answer for your question? I am in the same situation and I need help. Thank you $\endgroup$
    – LSola
    Apr 10, 2017 at 16:12

2 Answers 2

Reset to default
20
$\begingroup$

After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped research papers, as I was more interested in practical aspects of the topic.

Resources:

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ A number of these are resources I've been looking at - I've implemented a modified version of the work in points 2 and 4, for instance - so we're probably on the same-ish page now. And the vast majority of what I know is based on Eamonn Keogh's papers or articles based on them. But there are some here I hadn't read, and the one about bike share time series clustering is interesting - thanks! I'm not seeing anything that specifically answers my question, but do point it out if I've missed something while reading. $\endgroup$
    – Jo Douglass
    Dec 27, 2014 at 13:43
  • 1
    $\begingroup$ Also, if you're still finding this interesting, Keogh's papers are really worth a read. They're surprisingly easy to read and quite practical given the focus on using many data sets, and providing enough information that someone could re-create all experiments. The most recent one is interesting,and is what I was working my way through when I got sidelined by my question. cs.ucr.edu/~eamonn/selected_publications.htm $\endgroup$
    – Jo Douglass
    Dec 27, 2014 at 13:46
  • 1
    $\begingroup$ @JoDouglass: You're welcome! I didn't intend to answer your question directly (due to my limited knowledge of the topic), but hoped that it would be helpful, which appears to be the case. Thank you for nice comments and the reference - I will browse the papers and try to get a better idea. There is so much to learn, it's overwhelming a bit. $\endgroup$
    – Aleksandr Blekh
    Dec 27, 2014 at 16:05
  • 1
    $\begingroup$ Overwhelming is right, I was kicking myself for choosing this topic for a while! I feel like I'm getting there, though - and it's been really interesting to learn about. I have a number of things up and running as sort of rough versions of what I need to do, and I think it's more about figuring out how to process my data before running it through my models, now. That bike share link is interesting to me as it's the first I've seen discussing averaging of time series since reading the recent Keogh paper I mentioned. $\endgroup$
    – Jo Douglass
    Dec 27, 2014 at 16:30
  • 1
    $\begingroup$ @JoDouglass: When I said "overwhelming", I meant the whole data science domain (including AI/ML and statistics, specifically). I'm yet to find a resource, which presents a high-level discussion of various approaches and/or methods as themes, integrated into a comprehensive, yet parsimonious, framework. $\endgroup$
    – Aleksandr Blekh
    Dec 27, 2014 at 17:09
4
$\begingroup$

If you want to just mine for seasonal patterns, then look into autocorrelation. If you're looking for a model that can learn seasonal patterns and make forecasts from it, then Holt-Winters is a good start, and ARIMA would be a good thing to follow up with. Here[pdf] is the tutorial that got me off the ground.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The work is (for now) academic, rather than purely practical. I may do some forecasting very late on or in future, but I'm more interested in exploring the past data for now. The clustering is a goal in and of itself, as well as some ideas I want to explore past that point. $\endgroup$
    – Jo Douglass
    Dec 22, 2014 at 5:24
  • $\begingroup$ Sorry, hit enter prematurely. I've looked into autocorrelation to some extent and ran it on a subset of my data a whole ago but it wasn't really clear to me what I could get out of it. The data is pretty noisy. The seasonality patterns are sometimes pretty obvious on visualisation, but inexact in their timings - so I may be looking for similar patterns but not on a nice, even schedule. I was told that autocorrelation was likely to be problematic on such data, but happy to have another look if there's value in it. I don't want to just find seasonality, but understanding it is a goal. $\endgroup$
    – Jo Douglass
    Dec 22, 2014 at 5:47
  • $\begingroup$ Work through that tutorial at least up to and including 2.5. It uses R which is especially good for your academic environment. It will teach you autocorrelation which sounds like exactly what you're looking for (can't tell if it wasn't a fit because you didn't know what you were looking at, or the data is actually too noisy). If noise is the issue, exponential smoothing is one way to help with that, which will be taught as a part of the holt-winters model. Even if all of that doesn't give you the answer, it will certainly make your next step way clearer. $\endgroup$
    – TheGrimmScientist
    Dec 22, 2014 at 6:45
  • $\begingroup$ I had a read through the tutorial, but it mostly goes over things I already know. I'm actually working in Python and I'm a bit too far in to things to switch to R, although I've intended to grab rpy at some point in case there were some things I couldn't find in any Python libraries. I've re-written my question in case it helps any - like I say, the clustering is a goal in and of itself, I'm not looking for an entirely different direction to go in. I'm afraid the tutorial doesn't really answer my question. $\endgroup$
    – Jo Douglass
    Dec 25, 2014 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.