Timeline for Data preprocessing - Time Series data resets its values - Detection & Correction
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 13, 2021 at 11:27 | comment | added | Joschua Xner | i tried a few anomaly detection models, but nothing resulted in accaptable detections. My current solution is to compute the delta between the row t and t+1. If it's negative there's a reset and i can remove the timestamp via simple filter. | |
| Jun 11, 2021 at 18:14 | comment | added | Jon Nordby | each chunk can be processed independently, so it can be distributed among many workers in a cluster | |
| Jun 10, 2021 at 15:00 | comment | added | Joschua Xner | yeah, but i am using pandas mainly for small visualizations. It' doesnt really scale very well and can not be distributed among clusters. | |
| Jun 10, 2021 at 12:23 | comment | added | Oxbowerce |
If it is just about the number of lines you could relatively easily process the data in chunks of let's say 1 million in pandas, comparing for a single column if the current value is lower than the previous value using the .shift() method. Since pandas is built on top of numpy these operations are vectorized and really quick to compute.
|
|
| Jun 10, 2021 at 12:02 | comment | added | Joschua Xner | Not really. comparing 40 mil+ lines with each other on a high dimensionality basis isn't a very compute effective solution. | |
| Jun 10, 2021 at 11:18 | comment | added | Oxbowerce | Would a simple comparison between the current value and the previous value work? If the current value if lower than the previous value then the row is a reset, otherwise it is a continuation of the timeseries. | |
| Jun 10, 2021 at 10:29 | comment | added | Joschua Xner | Yes, the values are always lower then the values before and they don't always reset to zero. | |
| Jun 9, 2021 at 15:03 | comment | added | Oxbowerce | How do you define a reset, do the values always reset to zero? Are the values after a reset always lower then the values before the reset? | |
| Jun 9, 2021 at 13:37 | history | asked | Joschua Xner | CC BY-SA 4.0 |