# how to use CNN-LSTM with timedistributed

I am trying to use CNN-LSTM model with keras to reconstruct the time-series images, but now there are some weird problems. The input image is gray-scale and the input shape is (time_step, row, column, channel)=(4,64,64,1) and ouptput shape is also (time_step, row, column, channel)=(4,64,64,1). Pixel value is between 0~1.

Following is the code:

def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=TimeDistributed(SEResNet_model)(input_data)
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(1,1),activation='relu',name="conv_2"))(x)

output=TimeDistributed(Conv2D(1,kernel_size=
(1,1),activation='relu',name="conv_final"))(x)
model =Model(inputs=input_data, outputs=output)
return model


I do the SEResNet model and put it in the TimeDistributed. When I train this model, I find the loss is inf. I don't know why it happened, so I try several experiments.

The first experiment:I put a convolution layer instead of SEResNet model in the TimeDistributed. I find the loss is also inf.

Following is the code:

def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(1,1),activation='relu',name="conv_1"))(input_data)
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(1,1),activation='relu',name="conv_2"))(x)
output=TimeDistributed(Conv2D(1,kernel_size=
(1,1),activation='relu',name="conv_final"))(x)
model =Model(inputs=input_data, outputs=output)
return model


The second experiment:I put the SEResNet model behind the ConvLSTM2D layer. I train this model again. I find the loss is not inf anymore.

Following is the code:

def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(1,1),activation='relu',name="conv_2"))(x)
x=TimeDistributed(SEResNet_model)(x)
output=TimeDistributed(Conv2D(1,kernel_size=

adam = Adam(lr=LR,decay=1e-4,beta_1=0.9, beta_2=0.999, clipnorm=1.0)