# clarify convLSTM usage for regression

I am trying to use keras and convLSTM layer to predict future weather data based on previous weather radar pictures.

I use i timesteps i.e. i radar images as input data. In my diagram, I assume i=3. These radar images go through a convLSTM keras layer. The convLSTM layer parameters require an input shape of the form : (batch_size, time, channels, image_height, image_width)

question 1 : in keras, the convLSTM layer does not require a timestep argument. So I assume it infers the number of timesteps from the input_shape. Is my understanding correct ?

In my problem, I would like to predict j timesteps ahead. It is therefore my understanding that time parameter of input_shape should equal i+j and that I should pad my input data with "blank" radar images. Indeed, this seems the way to create a many to many convLSTM layer where I need to predict timesteps ahead. In my diagram I assumed j=3 and therefore i+j=6

question 2 : is the above understanding regarding a convLSTM layer shape correct ?

The capacity of the convLSTM layer will be defined by the numbers of filters inside each LSTM cell i.e. by the filters parameter of the keras convLSTM layer. For the sake of discussion, let's assume that the number of filters is defined by k. Therefore the output of each LSTM cell will be (batch_size, k, H, W) where H and W are computed according to the convolutional neural network stride S and padding P and filters dimensions F i.e.

H = image_height−F+2P/S+1

W = image_width−F+2P/S+1

The network structure would then be like this :

Main question: How should I use these outputs to perform the desired regression i.e. to predict j timesteps ahead weather data ?

There is the well-known picture of weather nowcasting :

but I can't understand how to implement this structure. Furthermore, the structure does not make sense to me based on the knowledge I have with classical deep neural network and convolutional neural network.

if I were to implement this kind of structure in keras, I would use the code herebelow, based on the data structure discussed here above. I have no reason for the batch normalization apart that I saw this implementation here . Question 3 If someone knows why batch normalization is useful, I am all ears As one can see, the proposed code stops after the second convLSTM layer as I don't know how to process the data to perform the regressions based on the features extracted by the convLSTM layers. Any help would be hugely appreciated.

# define Architecture
from keras.layers import Dense, Conv2D, ConvLSTM2D, Activation, MaxPooling2D, Flatten, Dropout, concatenate, Input, BatchNormalization
from keras.models import Model
from keras.models import Sequential
import keras_applications

targetNb = 8 # need to predict eight time steps ahead
availableTimeSteps = 3 # number of timesteps i.e. radar images used to predict the next ones
toPredictTimeSteps = 3
totalTimeSteps = availableTimeSteps+toPredictTimeSteps

inputConvLSTM = Input(shape=((totalTimeSteps , channels, image_height, image_width))) # samples first dimension not necessary
x = ConvLSTM2D(filters=32,  # in convLSTM, #filters defines the output space dimensions & the capacity of the network. Similar to #units in a LSTM
kernel_size=(3,3),
strides=(1, 1),
padding='same' # no reason for this setting, just saw it on keras example
return_sequences=True, # return_sequences defines if the output returns the last time step output or all the time steps)(inputConvLSTM)
# given that the #timestep is not specified, one assumes the #timesteps
# is implicitly defined by the input_shape

x = BatchNormalization()(x)
x = ConvLSTM2D(kernel_size=(3,3),
strides=(1, 1),
return_sequences=True,
name = 'finalConvLSTM')(x)

# HERE I AM STUCK
# I do not know how to use the features of the convLSTM layers to make
# predictions. In a classical deep convolutional neural network I would use a flatten so something like :

x = TimeDistributed(Flatten(), input_shape=(model.get_layer('finalConvLSTM').output_shape[1:])

# flatten each time step
# x shall have dimensions (samples, time, filters, output_row, output_col)
# where output_row and output_col are computed according to convolutional rules
# so the flatten() shall produce, for each time step, a output of dimensions
# (samples, time, filters*output_row*output_col)


I just want to stress an important point: ConvLSTM() layers have been excluded from the new TensorFlow 2.0, which is largely based on Keras in models' specification. It is substituted by ConvLSTM2D() layers, that take different arguments as input. (see docs here). (An alternative is to manually create a combination of Conv2D() and LSTM() layers.)

That is to say that ConvLSTM() layers might disappear in the not-too-distant future, and that studying how ConvLSTM2D() layers work might be a good investment.

However, coming to your questions:

in keras, the convLSTM layer does not require a timestep argument. So I assume it infers the number of timesteps from the input_shape. Is my understanding correct ?

When working with RNNs, you should specify the lenght of the input series (the number o ftimesteps) from within input_shape.

is the above understanding regarding a convLSTM layer shape correct ?

I'm not sure I understood what you're asking here. What do you mean with:

I should pad my input data with "blank" radar images. Indeed, this seems the way to create a many to many convLSTM layer where I need to predict timesteps ahead.

?

How should I use these outputs to perform the desired regression

You can predict:

• one value at a time, with one output node at the end of the model, and then iterate prediction
• many values at a time, with multiple
• output nodes use a seq2seq model, in which you predict a sequence of the same length of the input, but shifter forward of one o more steps

This is all up to your preferences/needs.

why batch normalization is useful

It's very useful between Convolutional and Dense layers, it's a regularization technique that helps activation functions work as expected, and allows the model to better capture non-linearities. However, I don't recommend using with Recurrent architectures, since the distortion of the output time series would decrease model's quality. (Please keep in mind this is just a personal opinion.)

The convLSTM layer parameters require an input shape of the form : (batch_size, time, channels, image_height, image_width)

question 1 : in keras, the convLSTM layer does not require a timestep argument. So I assume it infers the number of timesteps from the input_shape. Is my understanding correct?

The ConvLSTM layer in Keras does take a time argument. You've just stated the input shape in the line above which clearly has time as its second argument, so I'm a bit confused about your question here.

The ConvLSTM layer in Keras takes in a full sequence, performs the unrolling internally, and returns the final timestep's output (or an output for every timestep if return_sequences=True).

In reply to your second question: you have drawn a standard many-to-many RNN architecture, this is not what you want here. What you want is an encoder-decoder RNN architecture which will allow you to predict your $$j$$ timesteps ahead, where $$j$$ does not have to be equal to the input timesteps.

First of all, it's a bit vague what's your output dimension. Is it the forecast of a single step (j-timestep ahead) or a vector of multi-steps?

Question1: I guess time should be similar to vanilla LSTM, which is the i in your case (I need to refer other work to confirm this).

Main question: I think this is similar to the question of how to solve regression by CNN (as your output of the ConvLSTM2D is 3- or 4-d data). You can either pass the output to a stacked LSTM, or to CNN-pooling and flattened to dense layers.