2
$\begingroup$

I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence).

Because the time series is very long, I plan to extract shorter, partially-overlapping subsequences and use each of them as one training input for the LSTM.

In my train/validation/test split, do I have to use older subsequences for training and newer for validation and test? Or can I treat them as if they were independent samples and just randomly shuffle them, given that anyway the LSTM will start each subsequence with empty memory?

The reason I ask is because I noticed that, due to how the timeseries was collected, the first half contains mostly behaviour A while the second half mostly behaviour B. This would cause training to be mostly on A and testing mostly on B, which does not reflect the fact that, in production, the system will see both periods of predominant A and periods of predominant B.

$\endgroup$

2 Answers 2

1
$\begingroup$

[EDIT] In case you have very long sequences, you can also try attention-based models to prevent the vanishing gradient issue.


If I got it correctly, you might have the following cases:

  • labeled samples with tag A or B per date-time index, having as informative attributes the present + some lag values of interest --> (this would be a standard classification approach without time ordering bein necessary)

  • sliding window of samples (what you mean by subsequences) --> here you should respect the time ordering at least for the validation set, so you make sure you evaluate your LSTM with a real scenario with future never seen sequences.
    With this second approach, you can indeed shuffle the training batches (via the keras shuffle(BUFFER_SIZE).batch(BATCH_SIZE) functionality for instance (info here), but leaving the validation set without shuffling, as follows:

      BATCH_SIZE = 256
      BUFFER_SIZE = 10000
    
      train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi,y_train_multi))
      train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
    
      val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
      val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()
    

You can find a complete worked-out example here

You can also make use of the time series data preprocessing helper where you can decide whether to shuffle or not.

$\endgroup$
1
$\begingroup$

Cross-validation for time series is a very complex topic. My impression is that there is no final consensus on what the best approach is. For a quick reference, you can e.g. consult this thesis or the documentation related to the tscv library.

If you are dealing with an autoregressive problem, where independent variable and target coincide, I think the issue is slightly easier than the general case, though.

The way I see it, in the case of a neural network model, of the necessity for i.i.d. data boils down to using the mean gradient value over a batch. If your windows are not i.i.d., such mean will obviously be a biased estimator of the true gradient. Other than that, at prediction stage the process is purely deterministic. That is, 'past' or 'future' doesn't matter if you use a trained model to predict $t$+1 given the previous ($t-N$ .., $t$) timesteps. As a pre-emptive answer, one can imagine to flip the time series and re-fit, and the accuracy of the prediction should not be affected by the causality (assuming the causality dependency can be reversed).

It also helps thinking of the time series in terms of its Fourier transform. That one is actually free from the time dependency, and nobody would argue that you can use a FFT for drawing samples at any timestep $t$ along the sequence. So yes, in practice the points are causally connected, but from the point of view of the analysis, this caveat is only used in the creation of the windows and in the minimization step.

Again, this is only my interpretation and I stress that I only sustain it in the case of autoregressive tasks, using NNs, where there train/test windows do not overlap. I am open to other point of views, please leave me a comment if I overlooked some concept.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.