Training a LSTM on a time serie containing multiple inputs for each timestep

Question

I am trying to train a LSTM in order to use it for forecasting. The problem is basically a multivariate multi-step time series problem.

It is simply an experiment to see how statistical models (ARIMA, Holts-Winters, ...) and neural networks compare for a given problem.

As my dataset is perfectly fit for a statistical model, I am having trouble when trying to format it to train the LSTM as I have multiple entries for one timestep (corresponding to different entities) and I don't really know how to deal with it since the sequence is no longer tied by the time of observation. Let's say my dataset looks like the following example :

time | ent | obs 

  1 ---   1 ------      5 

  2 ---   1 ------     6

  2 ---   5 ------     1

  3 ---   2 ------     7

  3 ---   5 ------     4 

As you can see, not every entity have an entry for any given time, and one timestep can have multiple entries.

I thought of training the LSTM for each entity but I would have too few data for most of them. Some threads gave me the idea to separate each entity into batches but the number of observations is not constant so it wouldn't work for me.

How do you think I am supposed to tackle this problem?

Answer 1

The answer to this question highly depends on what relationship between the variables you are interested in.

• If you are interested in the relationship between time and observation-value, treating the entities as different batches could make sense, under the assumption that the role of individual entities doesn't really matter to you. In this case, you would, for example, add the mean of each entity (or the overall mean) to all entities with missing values to get a constant number of observations per entity. But you could also simply average all values in each timestamp and include other features as min & max. This would most probably deliver better results.

• If you are interested in the relationship between entities and observation-value, this is a matter of missing data in time series. There are a lot of techniques that can help you with that from simply imputing the mean to more sophisticated methods like a Kalman filter. However, in the end, you will have to ask yourself why these observations are missing and choose the appropriate method. But since you are using time-dependent models in your experiment, I assume, this is not of interest to you.

• If you are interested in the interrelationship of all three variables, you are dealing with panel data. In this case, I don't see a reasonable possibility to model this with an LSTM. Maybe another RNN-architecture could work, however, the only paper I found was Tensorial Recurrent Neural Networks for Longitudinal Data Analysis from Mingyuan et.al. But in the end, it would not matter, since an ARIMA-model also isn't appropriate for panel data. Usually, you use a Difference-In-Differences approach for that kind of data. In this case, I would suggest changing the dataset for your experiment.