Why does my LSTM perform better when randomizing training subset vs. standard batch training?

Question

I am training a simple LSTM network using Keras to predict time series values. It is a simple 2-layer LSTM.

I get the best performance when I train on subsets of the training set that start at random points. Each subset has a training size of 100 samples and a validation size of 30 samples. At each sample the model has a batch size of 16, trains for 100 epochs with an early stop after 20 epochs of little improvement. I run this training 10 times.

How is this different from me training me simply training my model as such:

model.compile(loss='mean_squared_error', optimizer='adam')

results = model.fit(X_train, y_train, epochs=100, batch_size=32, shuffle=False,
                validation_split=0.3, verbose = 1, steps_per_epoch=10,
callbacks = [EarlyStopping(monitor='val_loss', min_delta=5e-5, patience=20, verbose=1)])

Doesn't Keras automatically select random subsets of the training data for training and validation through the mini batches? Is it simply because I am training the model so many more times?

Is this effectively a k-fold-esque training implementation? However, using sk-learn's KFold or Repeated Kfold doesn't get results as good.

NB: I do not want to use shuffle as this is time series data and that would distort the model training.

Answer 1

Couple of points:

1) When you 'run this training ten times' does that mean you completely re-initialize the model and train from scratch?

2) I'm not entirely sure that you're loading the data correctly from the code you've supplied. What are the shapes of X_train and y_train?

3) It may depend on your data and your selection process. Time-series data is typically temporally correlated -- the closing value of the Dow Jones on Thursday is more likely to be close to the values on Wednesday and Friday than on Monday or in two weeks. It may be that if you sample 100 data points in a row and they're temporally correlated, you're giving your model an easier problem, because those values are more like each other than values elsewhere in the series.

Good luck!