0
$\begingroup$

1. Summarize the problem

I currently trying to work with time series data from sensors which has some problems regarding resetting it values. For example some cumulative values gets reset and don't add up the latest value.

| timestamp | col_01 | col_02 | col_03 |
|-----------|--------|--------|--------|
| 123456    | 0      | 0      | 0      |
| 123457    | 12,8   | 0,14   | 0      |
| 123458    | 85,7   | 3,87   | 1,5    |
| 123459    | 140,6  | 41,2   | 1,5    |
| 123460    | 210,8  | 78,1   | 5,7    |
| 123461    | 14,9   | 9,42   | 0,8    | <- reset
| 123462    | 60,2   | 12,47  | 1,9    |

2. Provide details and any research

The dataset consists of multiple GB of sensor data and checking every row for a reset and summing it up isn't a very efficient method and consumes a lot of time.

3. When appropriate, describe what you’ve tried

I tried using some outlier detection with the Z- & IQR-Score Method, but these approaches detect the wrong values. Mostly the points after the sensor data rested itself.

What's a suitable approach to solve this problem? Using a time series cycle detection to get the timestamps of the resets? The next step after detecting these outliers would be to correct or remove them from the dataset.

Improve this question
$\endgroup$
8
  • $\begingroup$ How do you define a reset, do the values always reset to zero? Are the values after a reset always lower then the values before the reset? $\endgroup$
    – Oxbowerce
    Commented Jun 9, 2021 at 15:03
  • $\begingroup$ Yes, the values are always lower then the values before and they don't always reset to zero. $\endgroup$
    – Joschua Xner
    Commented Jun 10, 2021 at 10:29
  • $\begingroup$ Would a simple comparison between the current value and the previous value work? If the current value if lower than the previous value then the row is a reset, otherwise it is a continuation of the timeseries. $\endgroup$
    – Oxbowerce
    Commented Jun 10, 2021 at 11:18
  • $\begingroup$ Not really. comparing 40 mil+ lines with each other on a high dimensionality basis isn't a very compute effective solution. $\endgroup$
    – Joschua Xner
    Commented Jun 10, 2021 at 12:02
  • $\begingroup$ If it is just about the number of lines you could relatively easily process the data in chunks of let's say 1 million in pandas, comparing for a single column if the current value is lower than the previous value using the .shift() method. Since pandas is built on top of numpy these operations are vectorized and really quick to compute. $\endgroup$
    – Oxbowerce
    Commented Jun 10, 2021 at 12:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.