LSTM for time series - which window size to use

Question

I have a LSTM based network which inputs a n-sized sequence of length (n x 300) and outputs the next single step (1 x 300).

The "raw" data consists of a few thousand semi-processed sequences of variable length where each step is (obviously) 1 x 300. Hence X is (n x 300).

The way I am generating the training and test data, from this semi-processed original sequences is:

1. drop all original sequences shorter than K=9
2. apply a sliding window with stride 1 and length K=9 to each original sequence kept
3. shuffle the generated data
4. separate train/dev test/test data

Now all training/testing data is [9 x 300] and Y is [1 x 300]

The resulting network starts overfitting around epoch 10 which led me to ¹. This is itself is not a problem since the results are of good enough quality.

If I try sequences as short as 6 (as in 6 x 300, that were dropped during the first phase) it gives proper enough information for us in y.

Hence the questions are about good practices and improving the network.

Technically, I am using Keras and using Google Cloud ML to execute the hyper parameter search.

The questions are:

1. should I stick to generating data with sliding windows since it is working properly?

2. if sliding window, k=9 is kind of a best guess, should I hyperparameterize this number K and search for the minimum loss with it in mind? I am afraid of doing so because hyperparameters typically affect the network itself, not the training data set, but note that K defines the data set length, would I be generating undesired bias by hyperparameterizing it?

3. if sliding window, should I be generating y shaped [9x300] with y=[x2, x3,...,x9, x10], instead of the current y generation? if I try both output structures and pick the one that lowers the loss in the dev test, I would check against a validation set?

4. if no sliding window, should I be training with one sequence per original time series input? and use attention or bidirectional lstm? ²

5. shuffling generates batches with shuffled partial sentences instead of batches from the same original sequence. but since shuffling is done prior to separating data for training and testing, it seems like I am not doing it the best way (or even properly). Should I first separate the original long sentences into train/each test phase, and then break them with the sliding windows (if applicable)?

References:

• ¹ sliding window leads to overfitting in LSTM?
• ² RNNs for time series prediction - what configurations would make sense

Answer 1

Three simple steps:

• Make data stationarity (remove trends and seasonality).
• Implement PACF analysis on the label data (For eg: Load).
• Interpret the PACF plots to choose an optimal lag value for sliding
  window

Concise tutorial for PACF analysis: http://rinterested.github.io/statistics/arima.html

Answer 2

In sentiment analysis it's common to decide a sentiment length (number of words in sentence) based on the mean of the trainset and do padding for shorter sentences and cut for longer.

I think this would be the best choice.

Answer 3

So your question is about the window size of LSTM. Selecting the window size depends on the dataset. For example, in the case of stock data, you may choose a big window size. I saw some papers of stock prediction where the window size is set up to 30.

Please note that if the big window size means we are working with a complex network. That means the training time also increases. Therefore you need to think about the accuracy and computation complexity of the model when selecting the window size.