I am trying to use CNN-LSTM model with keras to reconstruct the time-series images, but now there are some weird problems. The input image is gray-scale and the input shape is (time_step, row, column, channel)=(4,64,64,1)
and ouptput shape is also (time_step, row, column, channel)=(4,64,64,1)
. Pixel value is between 0~1.
Following is the code:
def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=TimeDistributed(SEResNet_model)(input_data)
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(3,3),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_2"))(x)
output=TimeDistributed(Conv2D(1,kernel_size=
(1,1),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_final"))(x)
model =Model(inputs=input_data, outputs=output)
return model
I do the SEResNet model and put it in the TimeDistributed. When I train this model, I find the loss is inf. I don't know why it happened, so I try several experiments.
The first experiment:I put a convolution layer instead of SEResNet model in the TimeDistributed. I find the loss is also inf.
Following is the code:
def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(3,3),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_1"))(input_data)
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(x)
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(3,3),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_2"))(x)
output=TimeDistributed(Conv2D(1,kernel_size=
(1,1),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_final"))(x)
model =Model(inputs=input_data, outputs=output)
return model
The second experiment:I put the SEResNet model behind the ConvLSTM2D layer. I train this model again. I find the loss is not inf anymore.
Following is the code:
def CNNLSTM_model():
nb_filter=32
input_data = Input(shape=(4,64,64,1),name="input")
x=ConvLSTM2D(filters=64, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True)(input_data)
x=Bidirectional(ConvLSTM2D(filters=32, kernel_size=(3, 3),
padding='same',activation='relu',recurrent_activation='relu',return_sequences=True))(x)
x=TimeDistributed(Conv2D(nb_filter,kernel_size=
(3,3),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_2"))(x)
x=TimeDistributed(SEResNet_model)(x)
output=TimeDistributed(Conv2D(1,kernel_size=
(1,1),kernel_initializer="he_normal",padding="same",strides=
(1,1),activation='relu',name="conv_final"))(x)
model =Model(inputs=input_data, outputs=output)
return model
What's wrong with the code? Thanks everyone.
adam = Adam(lr=LR,decay=1e-4,beta_1=0.9, beta_2=0.999, clipnorm=1.0)
model.compile(adam,
loss=['mse'],
metrics=[PSNR]
)