I have a streaming data along with timestamp dataset that looks like this: Sample Data Look alike1.png

Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).

Column1, Column2 .... ColumnN correspond to the variables (they change over time) and "Label" shows the different samples. You can assume that the values tend to decrease over time for a particular label.

Labels A1,B1,C1.........A2,... M labels.

Note : Values of timeNew of a Label depends on values of timeOld of that Label and Labels belong to its cluster.

I need to group Labels with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).

I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.

To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.

Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.

bumped to the homepage by Community yesterday

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

It depends a bit if the timestamps have any connection to each other (is t2 impacted by t1 as example).

In general this looks like a classification problem and you can use for example sklearn. If you want to distinguish A and C and all other cases you would end up with a multi-class classification problem and not all algorithms support these. If not You can just transform the Label into target (A+B) /non_target (the rest).

Three additional advice:

  1. You can use pandas Dataframes for the pre-processing.
  2. You should at least have a basic understanding of test and training samples before you start anything.
  3. The correct success metric is both important and sometimes challenging to find. Quick example: If you choose accuracy and 99% of your values belong to a class any algorithm that per default predicts this class will have a very high accuracy.

If timestamps are connected you might want to do some pre-processing and add data from previous timestamps to the current one (example: average value of column 1 of last x previous tstamps).

  • Apologies, i was not clear earlier. Please do read the question now. @el-burro – Abhinaya Krishna Mar 21 at 23:15

Aleksandr Blekh's answer in this older question provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):

Before proceeding with any method, I believe it is important to spend some time to think of the following:

  • Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better).

  • Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).

  • Depending on what you are trying to find out, you need to decide whether Dynamic time warping (DTW) is useful for you. For example, if 2 time-series have the exact same pattern but one of the 2 has a time-delay, should they still belong in the same cluster? (and how small/big time-delay is acceptable to put them in the same cluster?)

  • Yes, data generally changes over hours/days. I do agree with your statement that DTW might be useful. I'm confused about DTW to apply on the columns. as i can add columns like day of the week and stuff along with the given N columns. I wanna know how to approach with DTW when you have multiple columns. And there can many class labels like A,B,C,D...... I need to group them based on their similarity over time. @missrg – Abhinaya Krishna Mar 21 at 7:51
  • So if I understand correctly what you have is multivariate time-series, right? (several columns correspond to each label) One (simpler) approach would be to calculate similarity metric for each time-series separately (i.e. per column) and then sum them up before the clustering (e.g. using a weighted average). In any case, I will edit my reply and add some reading material for finding similarity among multivariate time-series. – missrg Mar 21 at 9:32
  • You can also have a look in this post (look at the comments under the question): datascience.stackexchange.com/questions/13445/… – missrg Mar 21 at 10:15
  • check with the description now. thanks @missrg – Abhinaya Krishna Mar 22 at 1:16

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.