preprocessing time sequence

Question

I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event.

The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas:

vector = pd.get_dummies(sr)

This part takes about 15 seconds.

(2) Using a sliding window of 10, i create samples and labels as follows; i iterate the vector from (1), i take Xt as the label and Xt:t-10 as the data.

Code:

X = np.zeros((len(samples), window_size, voc_len), dtype=np.bool)
y = np.zeros((len(samples), voc_len), dtype=np.bool)

if IN_COLAB:
  loading_bar = tqdm.tqdm_notebook(enumerate(samples),desc='Build dataset',total=len(samples))
else:
  loading_bar = tqdm.tqdm(enumerate(samples),desc='Build dataset',total=len(samples))

for numpy_index, pandas_idx in loading_bar:
  x_idx = (pandas_idx, pandas_idx + window_size)
  y_idx = pandas_idx + window_size
  Xt = vector.iloc[x_idx[0]:x_idx[1]]
  yt = vector.iloc[y_idx]
  X[numpy_index] = Xt
  y[numpy_index] = yt

display(yt)

The problem is that the seconds part is VERY slow (25 minutes). Is there a better way to do this? a built in function of some sort?

Thx

Answer 1

There are a couple of design patterns that are contributing to slow code:

1. Pandas is not designed for large-scale, fast data processing.

2. Your code is using a for loop which can be slow.

3. You are manifesting the sliding window before the program needs it. It might be better to create a view on the data and then manifest the data in memory only when the training needs it.

It would be better to use NumPy with numpy.lib.stride_tricks.sliding_window_view