How to arrange multiple multivariate time series of different length before passing it to Keras LSTM layer

Question

I have a number of multivariate time series that are produced by the same kind of process but:

• are of significantly different lengths;
• each time series is an independent instance, and the measurements are taken at different, quite random timestamps;
• each time series is related at every timestamp to two targets.

In other words:

• each time series has a shape of (n_timestamps, n_features)
• each target series has a shape of (n_timestamps, 2).

To give an example, this could be treated as stocks of different companies, that are described by few various features and the target at a given timestamp are probabilities that the final price at the end of the year will be higher than x, except we learn them directly from magically given ground-truth probabilities (instead of observed 0/1 responses).

I want to be able to predict the target at each time point and I wanted to give RNNs a try. However, I'm having issues with figuring out how I should arrange the data before passing it to Keras LSTM layers. The main things I'm wondering about are:

1. I want my RNN to use data starting from the beginning of the series to make prediction at time t, not only last k timestamps. I can't really use the whole history directly without exploding the gradient (it's too long), therefore I need a way to "remember" previously learned weights even though in reality my RNN will loop over last k timestamps.
2. Each time series has different length, so I'm unsure how to make things compatible with each other. I'm aware of padding as an option, but since the difference in length of examples can be as significant as 1000 vs 3000 this will results in many training examples that constitutes only of padding value.
3. Since measurements are taken at different timestamps, I believe it may affect my network in a sense that it can't really learn that e.g. last 10 timestamps are the most important. Or even if it can, these last 10 timestamps will have different lengths in reality for each input time-series... How big problem is this? Should I start with resampling all examples to the same time points (e.g. by interpolating)?

My current thinking is that:

• I can pad each of my example sequences to the same length (max(n_timestamps))
• Create batches of short sequences of length k, where k represents the length of the loop of RNN layer. In consequence, assuming I have 200 example sequences with the longest one has 3000 timestamps and my selected k is 50, it would result in 3000/50=60 batches of (200, 50) shape. Or should I make 3000-1 batches where one batch differs from the next one only by one timestamp (i.e. while the fist batch has timestamps from 1 to 50, the next batch has timestamps from 2 to 51 etc.)?
• Since padding was used, I would need to use Masking layer. Some (quite many) of the rows in prepared batches would constitute of inputs that should be ignored completely (as they would only have padding value for all 50 elements).

Is this the correct way to prepare the data for my problem? Can it be done better to not introduce bottlenecks such as learning using examples of only padding value (that should be ignored with masking layer). Or how can I prepare that data to address points 1., 2. and 3. described above?

Answer 1

To use multiple multivariate time series with different lengths and timestamps as input to a Keras LSTM model, you can follow these steps:

1. Pad the time series to the same length: You can pad each time series to the same length by adding zeros or a padding value at the end of the series. This will ensure that all the time series have the same shape and can be used as input to the LSTM model.

import numpy as np
import tensorflow as tf
import tensorflow.keras as keras

# Assume that `X` is a list of time series with shape (n_timestamps, n_features)
# and `y` is a list of target series with shape (n_timestamps, 2)

# Find the maximum length of the time series
max_length = max([X[i].shape[0] for i in range(len(X))])

# Pad the time series to the same length
X_padded = []
for i in range(len(X)):
  padded = np.pad(X[i], ((0, max_length - X[i].shape[0]), (0, 0)), 'constant')
  X_padded.append(padded)
X_padded = np.array(X_padded)

# Pad the target series to the same length
y_padded = []
for i in range(len(y)):
  padded = np.pad(y[i], ((0, max_length - y[i].shape[0]), (0, 0)), 'constant')
  y_padded.append(padded)
y_padded = np.array(y_padded)

2. Create batches of short sequences: You can create batches of short sequences by dividing each time series into overlapping or non-overlapping subsequences of a fixed length. For example, if you have 200 time series with the longest one having 3000 timestamps and you want to create sequences of length 50, you can create 60 batches of (200, 50) shape by dividing each time series into overlapping subsequences of length 50. Alternatively, you can create 2950 batches of (200, 1) shape by dividing each time series into non-overlapping subsequences of length 1.

# Create batches of short sequences by dividing each time series into overlapping subsequences of length 50
X_batches = []
y_batches = []
for i in range(0, max_length - 50 + 1, 50):
  X_batch = X_padded[:, i:i+50, :]
  y_batch = y_padded[:, i:i+50, :]
  X_batches.append(X_batch)
  y_batches.append(y_batch)
X_batches = np.array(X_batches)
y_batches = np.array(y_batches)

# Alternatively, you can create non-overlapping subsequences of length 1
# X_batches = X_padded[:, :-1, :]
# y_batches = y_padded[:, 1:, :]

3. Define the model: You can define the model using the Keras functional API by specifying the input and output layers. You can also add a Masking layer to ignore padded values in the input sequences.

# Define the model
inputs = keras.layers.Input(shape=(50, n_features))
masked = keras.layers.Masking(mask_value=0)(inputs)
lstm = keras.layers.LSTM(32)(masked)
outputs = keras.layers.Dense(2)(lstm)
model = keras.Model(inputs, outputs)

4. Compile and fit the model: You can compile the model by specifying the loss function, optimizer, and metrics. You can then fit the model on the batches of short sequences using the fit function.

# Compile and fit the model
model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])
model.fit(X_batches, y_batches, epochs=10, batch_size=32)

5. Predict on new data: You can use the predict function to get the predictions on new data. For example, to predict on a new time series of shape (n_timestamps, n_features), you can pad the time series to the same length as the training data, divide it into batches of short sequences using the same method as in step 2, and then use the predict function to get the predictions on each batch.

# Assume that `X_new` is a new time series of shape (n_timestamps, n_features)

# Pad the time series to the same length as the training data
X_new_padded = np.pad(X_new, ((0, max_length - X_new.shape[0]), (0, 0)), 'constant')

# Create batches of short sequences
X_new_batches = []
for i in range(0, max_length - 50 + 1, 50):
  X_new_batch = X_new_padded[:, i:i+50, :]
  X_new_batches.append(X_new_batch)
X_new_batches = np.array(X_new_batches)

# Predict on each batch
predictions = []
for X_batch in X_new_batches:
  y_pred = model.predict(X_batch)
  predictions.append(y_pred)
predictions = np.concatenate(predictions, axis=1)

Hope this is useful.

Answer 2

Giving a complete answer would be risky as there is a lot of unknown information.

Nevertheless, I would suggest starting with just one case, which is relatively easy (low noise and good quantity of data), reaching good results, and then repeating with other cases.

Then, I recommend Prophet for multivariate time series.

To have good predictions, I recommend using correlation maps to define the best variable inputs with each target output. Indeed, the more the features are correlated, the best predictions you will get. Keep in mind that you should keep anti-correlated data (around -1) because it is a form of correlation. In addition to that, some input features can be useless and would alter the predictions. Note that there would be one model per output.

For instance:

• If features A, C, E and F are (anti-)correlated together and with output O1, they would set the data for model M1.
• If features B, G and I are (anti-)correlated together and with output O2, they would set the data for model M1.
• If features D and H have no correlation with output 01 or output 02, they should be ignored.

Perhaps you should have independent models per each time series.

If some data is missing, applying an interpolation could be a good option.

If you want to solve predictions with multiple lengths with a single model, the best solution I know is to vectorize the time series, just like word2vec or other advanced NLP models. Time-series vectorization will help to have vectors representing various dynamics with various lengths.