0
$\begingroup$

I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and when data comes) based on breach of a static threshold value.

Question is related to the minutes fall within the period and do not have any recordings.

Let's assume today the sensor is recording values every 10 minutes (APPROXIMATELY) as shown below. What is the best possible way to handle the data in between the periodic interval. In below example, should I do a back fill with value 12 for all the points from 13:02 to 13:09?

2019-04-25T13:01:00  10,
2019-04-25T13:02:00  NAN,
2019-04-25T13:03:00  NAN,
2019-04-25T13:04:00  NAN,
2019-04-25T13:05:00  NAN,
2019-04-25T13:06:00  NAN,
2019-04-25T13:07:00  NAN,
2019-04-25T13:08:00  NAN,
2019-04-25T13:09:00  NAN,
2019-04-25T13:10:00  12,
2019-04-25T13:11:00  NAN,
2019-04-25T13:12:00  NAN,
2019-04-25T13:13:00  NAN,
2019-04-25T13:14:00  NAN,
2019-04-25T13:15:00  NAN,
2019-04-25T13:16:00  NAN,
2019-04-25T13:17:00  NAN,
2019-04-25T13:18:00  NAN,
2019-04-25T13:19:00  11,
2019-04-25T13:20:00  NAN
Improve this question
$\endgroup$
3
  • $\begingroup$ If you are processing in real time, and only thresholding, what point do you see in back filling at all? $\endgroup$
    – Stephen Rauch ♦
    Commented Apr 25, 2019 at 14:03
  • $\begingroup$ @StephenRauch There are 2 parts of use case - use case 1 is based on static threshold. Use case 2 (not mentioned before) is based on dynamic threshold whose value keeps on changing based on past data. This dynamic threshold is derived from mean (or exponential mean etc) of past values. This derivation requires me to get past data as well. Apologies for not mentioning this detail before. $\endgroup$
    – Siddharth vij
    Commented Apr 26, 2019 at 14:00
  • $\begingroup$ But why do you suppose that filling in data will help with computing a dynamic threshold? $\endgroup$
    – Stephen Rauch ♦
    Commented Apr 27, 2019 at 1:40

1 Answer 1

Reset to default
0
$\begingroup$

One option would be to interpolate the missing values. There are various methods depending on your use case (ie. linear, cubic, etc.)

https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html

You could also drop the missing values df.dropna() and just use the known values you have for thresholding.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

Improve this answer
$\endgroup$
1
  • $\begingroup$ what about the uneven time series..as you can see that the interval of data collection is not strictly 10 minutes. It can vary +-1 minute. $\endgroup$
    – Siddharth vij
    Commented Apr 26, 2019 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.