0
$\begingroup$

I first partition the timeseries data into train, validation, and test splits, without performing any shuffling.

Each row is a window of ordered samples, so my training data might be shaped (train_size, n_features, window_size).

I then shuffle and batch the samples, where each batch is shaped (batch_size, n_features, window_size). I am shuffling the rows and not their contents (the temporal ordering within each window is kept intact).

Why does shuffling yield better results than without shuffling?

$\endgroup$

1 Answer 1

1
$\begingroup$

Since you are training a neural network, the optimisation algorithm being used is gradient descent. Gradient descent works by taking small steps towards a solution, one minibatch at a time, which is why it expects the samples to be shuffled or independent and identically distributed (IID).

Shuffling means that each minibatch will contain a diverse set of samples, allowing the algorithm to iterate towards a general optimum that takes the diversity of the data into account. The relative independence between minibatches means the model is prevented from overfitting to the data's temporal order, and is encouraged to extract more generalisable patterns. The exposure to different patterns makes the net less likely to converge on a local minimum.

If, instead, the rows weren't shuffled, then each step in gradient descent could become strongly influenced by the temporal correlation with prior steps, and the net would end up focusing on the recent timepoints rather than seeing the bigger picture. This temporal correlation bias essentially means it forgets prior trends and becomes fixated on the most recent timepoints, and thus fails to generalise well to unseen data.

Shuffling therefore helps the model find more robust and generalisable trends, which is likely why your validation score is better.

Shuffling in this manner doesn't apply to stateful recurrent models though, as they assume the current minibatch starts exactly where the previous one ended.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.