I have a streaming data along with timestamp dataset that looks like this: Sample Data Look alike1.png

Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).

Column1, Column2 .... ColumnN correspond to the variables (they change over time) and "Label" shows the different samples. You can assume that the values tend to decrease over time for a particular label.

Labels A1,B1,C1.........A2,... M labels.

Note : Values of timeNew of a Label depends on values of timeOld of that Label and Labels belong to its cluster.

I need to group Labels with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).

I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.

To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.

Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.

  • $\begingroup$ @Vincenzo please look into this question now $\endgroup$ Commented Mar 21, 2018 at 23:24
  • $\begingroup$ @Toros91 refer the question now $\endgroup$ Commented Mar 21, 2018 at 23:25
  • $\begingroup$ I've retracted my vote, lets see what others think about it as well. $\endgroup$
    – Toros91
    Commented Mar 22, 2018 at 1:22
  • 1
    $\begingroup$ Looks like you are looking for a Multivariate Time series forecasting. $\endgroup$ Commented Nov 20, 2018 at 18:38

3 Answers 3


Aleksandr Blekh's answer in this older question provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):

Before proceeding with any method, I believe it is important to spend some time to think of the following:

  • Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better).

  • Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).

  • Depending on what you are trying to find out, you need to decide whether Dynamic time warping (DTW) is useful for you. For example, if 2 time-series have the exact same pattern but one of the 2 has a time-delay, should they still belong in the same cluster? (and how small/big time-delay is acceptable to put them in the same cluster?)

  • $\begingroup$ Yes, data generally changes over hours/days. I do agree with your statement that DTW might be useful. I'm confused about DTW to apply on the columns. as i can add columns like day of the week and stuff along with the given N columns. I wanna know how to approach with DTW when you have multiple columns. And there can many class labels like A,B,C,D...... I need to group them based on their similarity over time. @missrg $\endgroup$ Commented Mar 21, 2018 at 7:51
  • $\begingroup$ So if I understand correctly what you have is multivariate time-series, right? (several columns correspond to each label) One (simpler) approach would be to calculate similarity metric for each time-series separately (i.e. per column) and then sum them up before the clustering (e.g. using a weighted average). In any case, I will edit my reply and add some reading material for finding similarity among multivariate time-series. $\endgroup$
    – missrg
    Commented Mar 21, 2018 at 9:32
  • $\begingroup$ You can also have a look in this post (look at the comments under the question): datascience.stackexchange.com/questions/13445/… $\endgroup$
    – missrg
    Commented Mar 21, 2018 at 10:15

DTW per-group is the obvious answer. I've found DTW to be extremely computationally expensive, however.

My favorite technique in the world is lesser-known/used, and would handle your problem nicely, however... and have the added benefit of being scalable.

The downside to this technique is that it would only cluster on the "shape" of the time series, and not require it to line up at a specific time interval like DTW will.

It's called SAX. Basically, you represent the time series as a string of letters. Then, you can treat this string of letters exactly like you do in NLP or text-mining - by creating a frequency matrix for each letter, n-grams, etc. Now, along with all of those features you could add your other features as well, and run normal dimensionality reduction and clustering.


It depends a bit if the timestamps have any connection to each other (is t2 impacted by t1 as example).

In general this looks like a classification problem and you can use for example sklearn. If you want to distinguish A and C and all other cases you would end up with a multi-class classification problem and not all algorithms support these. If not You can just transform the Label into target (A+B) /non_target (the rest).

Three additional advice:

  1. You can use pandas Dataframes for the pre-processing.
  2. You should at least have a basic understanding of test and training samples before you start anything.
  3. The correct success metric is both important and sometimes challenging to find. Quick example: If you choose accuracy and 99% of your values belong to a class any algorithm that per default predicts this class will have a very high accuracy.

If timestamps are connected you might want to do some pre-processing and add data from previous timestamps to the current one (example: average value of column 1 of last x previous tstamps).

  • $\begingroup$ Apologies, i was not clear earlier. Please do read the question now. @el-burro $\endgroup$ Commented Mar 21, 2018 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.