4
$\begingroup$

I have a recurrent neural network with LSTM units that I want to train with batches of 6 timesteps. Each timestep is a record of a dataset and represents the temporal aggregation over 5 minutes of data taken every 10 seconds. Unfortunately, the dataset has temporal interruptions, so sometimes two successive records in the dataset can be temporally separated by 10 minutes to a few weeks. I was wondering if the timesteps I give to the network in each batch must all be separated by the same temporal distance, that is in my case they must all be within 5 minutes of each other or it is sufficient that they are subsequent and therefore I could also give the network for example a batch with 4 records at a distance of 5 minutes from each other and then 2 taken for example two days later.

UPDATED: The data comes from an electronic system so the time interruptions are due to the device being turned off. Rarely it is turned off only for a few minutes while most of the time it remains off for longer periods, like hours or sometimes days. Among other elements, the device includes a battery, therefore many of the features present in the dataset records are temperature, current and voltage readings which change value during temporal interruptions (little when turned off for a few minutes and significantly when turned off for hours or days). The goal of the network is to predict the value of some features two steps ahead.

| improve this question | | | | |
$\endgroup$
2
$\begingroup$

As always in ML modelling problems: it depends.

The critical factor here is that you are predicting based on properties of a sequence. The sequence does not need to be sampled at fixed time steps, or even be time based at all. E.g. in natural language processing the sequence of letters or words is only very loosely associated with the timing of the same items when spoken or read. The sequence does need to be drawn from a self-consistent source where moving from one item to next item in the sequence has the same meaning with respect to what you are trying to predict.

If the interruptions you see would cause large differences in the expected distribution of your input vector X or output Y, then you may not be able to use them as-is. You may still be able to rescue them as input data for a RNN, and make predictions based on them, if you add whether or not there has been an interruption - and if it might be relevant to your purpose, for how long - as additional features of input items in X.

Another option is to follow the time sequence strictly but add placeholder value of X (with a different flag set) for "unkown", maybe along with some rough guess for the missing features (or even a ML prediction from a different model), which you might do if the quantity being aggregated over is still active and important where there are missing records, but the aggregation service you are using as input has failed.

Depending on how much data you have available, and whether you need to make predictions at times when your aggregation service has failed, you could do anything from discarding incomplete data to ignoring the time differences, to interpolating missing records.

The first question you need to answer is "what does it mean for a sequence item to be missing":

  • If the whole system that provided the aggregation data is switched off, and there is no concept of pending items that will enter the system the moment it is switched on, then you can probably safely ignore the missing records and ignore the precise timing.

  • If your prediction target is sensitive to elapsed time, and yo uneed to make predictions even when there have been interruptions to the aggregation system, then you will want to augment the sequence data somehow to account for the missing records, and train/predict with that added feature engineering.

  • If only the aggregation is switched off (or has a chance of failure causing gaps in the record), and you don't need to run any predictions when it is not available, then you may be able to make the rule that the prediction system will only function when there are 6 recent consecutive records available, and discard the data that is causing you to worry. This makes sense if interruptions are rare or the predictive model is not critical.

If you are not sure which of these things to do, then the next approach is to do all of them that seem that they could be intuitively correct, tuning each model separately, then see how each different approach performs on your test set.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ I'm sorry, some information was missing, I updated the question. I don't need to run any predictions when the system is not available, so, if I understand correctly, in my case it would be good to make the system work only with 6 consecutive records in each batch? $\endgroup$ – Ingen 77 Sep 5 '19 at 13:58
  • $\begingroup$ @Ingen77: If you are predicting temperature readings, it doesn't make sense to use really old readings in the same sequence when there is a gap.Your options are bascially (a) to make predictions using shorter sequences sometimes (perhaps by filling in missing values, perhaps by allowing the RNN to predict from fewer readings, accepting that it may be less accurate) or (b) only start making predictions when you have enough sequence data on each current run. $\endgroup$ – Neil Slater Sep 5 '19 at 14:40
  • $\begingroup$ Yes, it doesn't make much sense to use such old values for these type of predictions. Thanks for the detailed answers! $\endgroup$ – Ingen 77 Sep 5 '19 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.