0
$\begingroup$

I have a timeseries stored as a pandas.Series, and have computed the autocorrelation to be negiligable for 12 hours lag. Therefore, I want to test the approach of sampling data points randomly, but with at least 12 hours between them, and then treating them as IID data.

How do I do this algorithmically, in an efficient way?


ps if I want at most one data point per 12 hour window, I could do like below:

import pandas as pd
import sklearn
import numpy as np

ndata = 2*24*60*60
s = pd.Series(data=np.random.random(ndata),index=pd.date_range(start=pd.Timestamp("2000-01-01"), periods=ndata, freq='S'),name='data')
window = pd.Series((s.index - s.index[0]).total_seconds()//(12*60*60),index=s.index,name='window_id')
df = pd.concat([s,window],axis=1)

random_datapoints = sklearn.utils.resample(df,stratify=df.window_id,n_samples=df.window_id.unique().size,replace=False)

However, most probably (this happens almost surely as $n\to\infty$, Ill get two samples that are less than 12 hours apart, even though they will be 12 hours apart on average....

$\endgroup$
1
$\begingroup$

Let's say you have very frequent data across a period of time T and you want to sample N points.

Instead of sampling points directly you could sample the time that separate them. You then just need to add 12 hours to all gaps to enforce your constraints.

To do so you could sample N points uniformly in [0, T - (N-1) x 12h] and then compute the difference between consecutive points.

Sampling N + 1 points in [0, T - N x 12h] allows you to also sample the first timestep.

Afterwards you can match thoses times with the closest samples available in your data.

import numpy as np
T = 500
N = 200
min_diff = 0.5

t_diffs = min_diff + np.diff(np.sort(np.random.uniform(0, T - N * min_diff, N+1)))

times = np.cumsum(t_diffs)

import matplotlib.pyplot as plt
plt.figure()
plt.scatter(np.arange(len(times)), times)
plt.ylabel("Time")
plt.xlabel("Point index")
plt.savefig("fig.png")
plt.show()

enter image description here

$\endgroup$
  • $\begingroup$ Nice. Simple and clear. :) $\endgroup$ – LudvigH Nov 6 '20 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.