Pre-processing irregular, high frequency time-series data in python

Question

...posted originally in StackOverflow (might be better suited here)

• Small Picture:

I am working on pre-processing irregular, high frequency time-series data. In one second, I can have multiple data points, as seen below in the timestamp field:

"timestamp": "2018-06-03T12:27:54.253" 
"timestamp": "2018-06-03T12:27:54.409"
"timestamp": "2018-06-03T12:27:54.548"

I'm in the process of developing a sampling scheme for this time-series data so that I can reduce the number of data points, and standardize time steps without losing information or introducing any bias.

So far I've been using Pandas pd.resample() on just a small subset of our data (5 days ~ 2 million records) by using mean as the aggregation function and linear interpolation. I am on downsampling the data by seconds, minutes, and hours for experimental purposes which takes care of the irregular time steps of the original data.

• Larger Picture:

I am working with many millions of records (data from April - today) that have been queried from elasticsearch and, ideally, want to pre-process/sample within this large pool of time series data to only obtain statistically significant data points. The purpose of this pre-processing step is for future data exploration/modeling.

• Question:

How can I modify my current pre-processing scheme to ensure optimal processing for Millions of records and still maintain their statistical properties (no bias introduced)?

I'm aware that there is possibly a better way to approach the processing this kind of data. Any input is greatly appreciated...thanks!

Answer 1

Keep in mind, downsampling is always lossy. I will give you two hints.

1. Nyquist-Shannon Theorem. What it says in short is that the max frequency that you may observe in a sampled signal is half of the sampling frequency. This means that if you downsample the data every 1 hour, you can observe dynamic phenomena with period of 2 hours (or more). Another way of looking at the theorem is "you have to sample the signal at least twice as fast as the maximum frequency that you expect to see in it".

   Engineering rule of thumb: Sample the signal at least 10 times faster than the expected highest frequency, not just 2 times.

2. Histogram Representation. Since you have a huge database, it might make sense to convert your variables into histograms! In this way, you will greatly reduce their size. You discretize every real-valued variable into bins and then instead of storing the exact value of the variable you just increase the count of the appropriate bin by 1. There are a lot of Machine Learning algorithms that deal with histogram representations, such as this:

   https://pdfs.semanticscholar.org/b8c8/347f9c33935b97703ecd35a67af5c5508487.pdf

Hope it helps :) !