1
$\begingroup$

I am trying to use CNN-LSTM model with keras to reconstruct the time-series images, but now there are some weird problems. The input image is gray-scale and the input shape is (time_step, row, column, channel)=(4,64,64,1) and ouptput shape is also (time_step, row, column, channel)=(4,64,64,1). Pixel value is between 0~1.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=TimeDistributed(SEResNet_model)(input_data)
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

I do the SEResNet model and put it in the TimeDistributed. When I train this model, I find the loss is inf. I don't know why it happened, so I try several experiments.

The first experiment:I put a convolution layer instead of SEResNet model in the TimeDistributed. I find the loss is also inf.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=TimeDistributed(Conv2D(nb_filter,kernel_size= 
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_1"))(input_data)
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

The second experiment:I put the SEResNet model behind the ConvLSTM2D layer. I train this model again. I find the loss is not inf anymore.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(input_data)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    x=TimeDistributed(SEResNet_model)(x)
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

What's wrong with the code? Thanks everyone.

adam = Adam(lr=LR,decay=1e-4,beta_1=0.9, beta_2=0.999, clipnorm=1.0) 
model.compile(adam,
                  loss=['mse'],
                  metrics=[PSNR]
                )
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.