I am trying to use keras and convLSTM layer to predict future weather data based on previous weather radar pictures.
I use i timesteps i.e. i radar images as input data. In my diagram, I assume i=3. These radar images go through a convLSTM keras layer. The convLSTM layer parameters require an input shape of the form : (batch_size, time, channels, image_height, image_width)
question 1 : in keras, the convLSTM layer does not require a timestep argument. So I assume it infers the number of timesteps from the input_shape. Is my understanding correct ?
In my problem, I would like to predict j timesteps ahead. It is therefore my understanding that time parameter of input_shape should equal i+j and that I should pad my input data with "blank" radar images. Indeed, this seems the way to create a many to many convLSTM layer where I need to predict timesteps ahead. In my diagram I assumed j=3 and therefore i+j=6
question 2 : is the above understanding regarding a convLSTM layer shape correct ?
The capacity of the convLSTM layer will be defined by the numbers of filters inside each LSTM cell i.e. by the filters parameter of the keras convLSTM layer. For the sake of discussion, let's assume that the number of filters is defined by k. Therefore the output of each LSTM cell will be (batch_size, k, H, W) where H and W are computed according to the convolutional neural network stride S and padding P and filters dimensions F i.e.
H = image_height−F+2P/S+1
W = image_width−F+2P/S+1
Main question: How should I use these outputs to perform the desired regression i.e. to predict j timesteps ahead weather data ?
There is the well-known picture of weather nowcasting :
but I can't understand how to implement this structure. Furthermore, the structure does not make sense to me based on the knowledge I have with classical deep neural network and convolutional neural network.
if I were to implement this kind of structure in keras, I would use the code herebelow, based on the data structure discussed here above. I have no reason for the batch normalization apart that I saw this implementation here . Question 3 If someone knows why batch normalization is useful, I am all ears As one can see, the proposed code stops after the second convLSTM layer as I don't know how to process the data to perform the regressions based on the features extracted by the convLSTM layers. Any help would be hugely appreciated.
# define Architecture from keras.layers import Dense, Conv2D, ConvLSTM2D, Activation, MaxPooling2D, Flatten, Dropout, concatenate, Input, BatchNormalization from keras.models import Model from keras.models import Sequential import keras_applications targetNb = 8 # need to predict eight time steps ahead availableTimeSteps = 3 # number of timesteps i.e. radar images used to predict the next ones toPredictTimeSteps = 3 totalTimeSteps = availableTimeSteps+toPredictTimeSteps inputConvLSTM = Input(shape=((totalTimeSteps , channels, image_height, image_width))) # samples first dimension not necessary x = ConvLSTM2D(filters=32, # in convLSTM, #filters defines the output space dimensions & the capacity of the network. Similar to #units in a LSTM kernel_size=(3,3), strides=(1, 1), padding='same' # no reason for this setting, just saw it on keras example return_sequences=True, # return_sequences defines if the output returns the last time step output or all the time steps)(inputConvLSTM) # given that the #timestep is not specified, one assumes the #timesteps # is implicitly defined by the input_shape x = BatchNormalization()(x) x = ConvLSTM2D(kernel_size=(3,3), strides=(1, 1), padding='same' return_sequences=True, name = 'finalConvLSTM')(x) # HERE I AM STUCK # I do not know how to use the features of the convLSTM layers to make # predictions. In a classical deep convolutional neural network I would use a flatten so something like : x = TimeDistributed(Flatten(), input_shape=(model.get_layer('finalConvLSTM').output_shape[1:]) # flatten each time step # x shall have dimensions (samples, time, filters, output_row, output_col) # where output_row and output_col are computed according to convolutional rules # so the flatten() shall produce, for each time step, a output of dimensions # (samples, time, filters*output_row*output_col)