# Pre-processing irregular, high frequency time-series data in python

...posted originally in StackOverflow (might be better suited here)

• Small Picture:

I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp field:

"timestamp": "2018-06-03T12:27:54.253"
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"


I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.

So far I've been using Pandas pd.resample() on just a small subset of our data (5 days ~ 2 million records) by using mean as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.

• Larger Picture:

I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling.

• Question:

How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)?

I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!