1
$\begingroup$

I am trying to use CNN-LSTM model with keras to reconstruct the time-series images, but now there are some weird problems. The input image is gray-scale and the input shape is (time_step, row, column, channel)=(4,64,64,1) and ouptput shape is also (time_step, row, column, channel)=(4,64,64,1). Pixel value is between 0~1.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=TimeDistributed(SEResNet_model)(input_data)
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

I do the SEResNet model and put it in the TimeDistributed. When I train this model, I find the loss is inf. I don't know why it happened, so I try several experiments.

The first experiment:I put a convolution layer instead of SEResNet model in the TimeDistributed. I find the loss is also inf.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=TimeDistributed(Conv2D(nb_filter,kernel_size= 
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_1"))(input_data)
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

The second experiment:I put the SEResNet model behind the ConvLSTM2D layer. I train this model again. I find the loss is not inf anymore.

Following is the code:

def CNNLSTM_model():
    nb_filter=32
    input_data = Input(shape=(4,64,64,1),name="input")
    x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(input_data)
    x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                 padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
    x=TimeDistributed(Conv2D(nb_filter,kernel_size=
                 (3,3),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_2"))(x)
    x=TimeDistributed(SEResNet_model)(x)
    output=TimeDistributed(Conv2D(1,kernel_size= 
                 (1,1),kernel_initializer="he_normal",padding="same",strides= 
                 (1,1),activation='relu',name="conv_final"))(x)
    model =Model(inputs=input_data, outputs=output)
    return model

What's wrong with the code? Thanks everyone.

adam = Adam(lr=LR,decay=1e-4,beta_1=0.9, beta_2=0.999, clipnorm=1.0) 
model.compile(adam,
                  loss=['mse'],
                  metrics=[PSNR]
                )
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.