4
$\begingroup$

I have a streaming data along with timestamp dataset that looks like this: Sample Data Look alike1.png

Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the previous values(rows i.e data which came earlier w.r.t time).

Column1, Column2 .... ColumnN correspond to the variables (they change over time) and "Label" shows the different samples. You can assume that the values tend to decrease over time for a particular label.

Labels A1,B1,C1.........A2,... M labels.

Note : Values of timeNew of a Label depends on values of timeOld of that Label and Labels belong to its cluster.

I need to group Labels with similar behavior over time together (e.g. Label A1 and Label C1 should be put in the same cluster and B1,D2 may fall into same cluster over time as they tend to behave similar over time).

I thought of using DTW and get the similarity of each Label with respect to other Labels. but not sure, how to proceed when i have N Columns.

To be precise, i need to group Labels based on their similarities (Column1 .. ColumnN) over time and group them.

Once i group them when new data comes in i should be able to predict the values(Column1.. ColumnN) for a Label based on the previously seen data(can be just minutes closer to the current prediction) and the values associated with the Labels in its cluster and predict it accordingly.

$\endgroup$
  • $\begingroup$ @Vincenzo please look into this question now $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:24
  • $\begingroup$ @Toros91 refer the question now $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:25
  • $\begingroup$ @Stephen refer the question now Thanks. $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:25
  • $\begingroup$ @Siong refer the question now Thanks. $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:25
  • $\begingroup$ @Thye refer the question now Thanks. $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:25
0
$\begingroup$

It depends a bit if the timestamps have any connection to each other (is t2 impacted by t1 as example).

In general this looks like a classification problem and you can use for example sklearn. If you want to distinguish A and C and all other cases you would end up with a multi-class classification problem and not all algorithms support these. If not You can just transform the Label into target (A+B) /non_target (the rest).

Three additional advice:

  1. You can use pandas Dataframes for the pre-processing.
  2. You should at least have a basic understanding of test and training samples before you start anything.
  3. The correct success metric is both important and sometimes challenging to find. Quick example: If you choose accuracy and 99% of your values belong to a class any algorithm that per default predicts this class will have a very high accuracy.

If timestamps are connected you might want to do some pre-processing and add data from previous timestamps to the current one (example: average value of column 1 of last x previous tstamps).

$\endgroup$
  • $\begingroup$ Apologies, i was not clear earlier. Please do read the question now. @el-burro $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 23:15
0
$\begingroup$

Aleksandr Blekh's answer in this older question provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):

Before proceeding with any method, I believe it is important to spend some time to think of the following:

  • Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better).

  • Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).

  • Depending on what you are trying to find out, you need to decide whether Dynamic time warping (DTW) is useful for you. For example, if 2 time-series have the exact same pattern but one of the 2 has a time-delay, should they still belong in the same cluster? (and how small/big time-delay is acceptable to put them in the same cluster?)

$\endgroup$
  • $\begingroup$ Yes, data generally changes over hours/days. I do agree with your statement that DTW might be useful. I'm confused about DTW to apply on the columns. as i can add columns like day of the week and stuff along with the given N columns. I wanna know how to approach with DTW when you have multiple columns. And there can many class labels like A,B,C,D...... I need to group them based on their similarity over time. @missrg $\endgroup$ – Abhinaya Krishna Mar 21 '18 at 7:51
  • $\begingroup$ So if I understand correctly what you have is multivariate time-series, right? (several columns correspond to each label) One (simpler) approach would be to calculate similarity metric for each time-series separately (i.e. per column) and then sum them up before the clustering (e.g. using a weighted average). In any case, I will edit my reply and add some reading material for finding similarity among multivariate time-series. $\endgroup$ – missrg Mar 21 '18 at 9:32
  • $\begingroup$ You can also have a look in this post (look at the comments under the question): datascience.stackexchange.com/questions/13445/… $\endgroup$ – missrg Mar 21 '18 at 10:15
  • $\begingroup$ check with the description now. thanks @missrg $\endgroup$ – Abhinaya Krishna Mar 22 '18 at 1:16

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.