I have a long list of event (400 unique events, sequence ~10M long). I want to train an RNN to predict next event.

The preprocessing steps i took are: (1) turning to OneHotEncoding using pandas:

vector = pd.get_dummies(sr)

This part takes about 15 seconds.

(2) Using a sliding window of 10, i create samples and labels as follows; i iterate the vector from (1), i take Xt as the label and Xt:t-10 as the data.


X = np.zeros((len(samples), window_size, voc_len), dtype=np.bool)
y = np.zeros((len(samples), voc_len), dtype=np.bool)

  loading_bar = tqdm.tqdm_notebook(enumerate(samples),desc='Build dataset',total=len(samples))
  loading_bar = tqdm.tqdm(enumerate(samples),desc='Build dataset',total=len(samples))

for numpy_index, pandas_idx in loading_bar:
  x_idx = (pandas_idx, pandas_idx + window_size)
  y_idx = pandas_idx + window_size
  Xt = vector.iloc[x_idx[0]:x_idx[1]]
  yt = vector.iloc[y_idx]
  X[numpy_index] = Xt
  y[numpy_index] = yt


The problem is that the seconds part is VERY slow (25 minutes). Is there a better way to do this? a built in function of some sort?



1 Answer 1


There are a couple of design patterns that are contributing to slow code:

  1. Pandas is not designed for large-scale, fast data processing.

  2. Your code is using a for loop which can be slow.

  3. You are manifesting the sliding window before the program needs it. It might be better to create a view on the data and then manifest the data in memory only when the training needs it.

It would be better to use NumPy with numpy.lib.stride_tricks.sliding_window_view


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.