...posted originally in StackOverflow (might be better suited here)
- Small Picture:
I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp
field:
"timestamp": "2018-06-03T12:27:54.253"
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"
I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.
So far I've been using Pandas pd.resample()
on just a small subset of our data (5 days ~ 2 million records) by using mean
as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.
- Larger Picture:
I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling.
- Question:
How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)?
I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!