0
$\begingroup$

I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and when data comes) based on breach of a static threshold value.

Question is related to the minutes fall within the period and do not have any recordings.


Let's assume today the sensor is recording values every 10 minutes (APPROXIMATELY) as shown below. What is the best possible way to handle the data in between the periodic interval. In below example, should I do a back fill with value 12 for all the points from 13:02 to 13:09?

2019-04-25T13:01:00  10,
2019-04-25T13:02:00  NAN,
2019-04-25T13:03:00  NAN,
2019-04-25T13:04:00  NAN,
2019-04-25T13:05:00  NAN,
2019-04-25T13:06:00  NAN,
2019-04-25T13:07:00  NAN,
2019-04-25T13:08:00  NAN,
2019-04-25T13:09:00  NAN,
2019-04-25T13:10:00  12,
2019-04-25T13:11:00  NAN,
2019-04-25T13:12:00  NAN,
2019-04-25T13:13:00  NAN,
2019-04-25T13:14:00  NAN,
2019-04-25T13:15:00  NAN,
2019-04-25T13:16:00  NAN,
2019-04-25T13:17:00  NAN,
2019-04-25T13:18:00  NAN,
2019-04-25T13:19:00  11,
2019-04-25T13:20:00  NAN
$\endgroup$
  • $\begingroup$ If you are processing in real time, and only thresholding, what point do you see in back filling at all? $\endgroup$ – Stephen Rauch Apr 25 at 14:03
  • $\begingroup$ @StephenRauch There are 2 parts of use case - use case 1 is based on static threshold. Use case 2 (not mentioned before) is based on dynamic threshold whose value keeps on changing based on past data. This dynamic threshold is derived from mean (or exponential mean etc) of past values. This derivation requires me to get past data as well. Apologies for not mentioning this detail before. $\endgroup$ – Siddharth vij Apr 26 at 14:00
  • $\begingroup$ But why do you suppose that filling in data will help with computing a dynamic threshold? $\endgroup$ – Stephen Rauch Apr 27 at 1:40
0
$\begingroup$

One option would be to interpolate the missing values. There are various methods depending on your use case (ie. linear, cubic, etc.)

https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html

You could also drop the missing values df.dropna() and just use the known values you have for thresholding.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

$\endgroup$
  • $\begingroup$ what about the uneven time series..as you can see that the interval of data collection is not strictly 10 minutes. It can vary +-1 minute. $\endgroup$ – Siddharth vij Apr 26 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.