why is mse training drastically different from the begining of each training with Encoder-Decoder

Question

I am using encoder-decoder model to predict binary images from grayscale images. Here is the model

inputs = Input(shape=(height, width, 1))

conv1 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)

conv1 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)

pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)

conv2 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)

conv2 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)

drop2 = Dropout(0.2)(conv2)

pool2 = MaxPooling2D(pool_size=(2, 2))(drop2)

conv4 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)

conv4 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)

drop4 = Dropout(0.2)(conv4)

pool4 = MaxPooling2D(pool_size=(2, 2))(drop4)

conv5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)

conv5 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv5)

drop5 = Dropout(0.2)(conv5)

up6 = Conv2D(16, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop5))

merge6 = concatenate([drop4,up6], axis = 3)

conv6 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)

conv6 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)

up7 = Conv2D(8, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv6))

merge7 = concatenate([drop2,up7], axis = 3)

conv7 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)

conv7 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)


up9 = Conv2D(4, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))

merge9 = concatenate([conv1,up9], axis = 3)

conv9 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)

conv9 = Conv2D(4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)

conv9 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)

conv10 = Conv2D(1, 1, activation = 'sigmoid')(conv9)

model = Model(inputs=inputs, outputs=conv10)
nadam = optimizers.Nadam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=None,  schedule_decay=0.0004)
model.compile(loss='mean_squared_error', optimizer=nadam, metrics=['accuracy', norm_mse])

nb_epoch = 30
batch_size = 8
history = unet.fit(training, trainingLabel,
      validation_data=(validation, validationLabel),
      batch_size=batch_size, epochs=nb_epoch, shuffle="batch", callbacks=[checkpointer],  verbose=1)

I also have training dataset size is: 6912, validation dataset size is: 1728 Each time I start training my encoder-decoder in the very beginning I get different training accuracy and normalized MSE. It significantly affects the testing each time. I understand that weights in the beginning are randomly chosen and so it will affect the performance on the first epoch. My concern is that the difference is too large. Here are some examples:

Epoch 1/30
6912/6912 [==============================] - 27s 4ms/step - loss: 0.0612 -    acc: 0.9252 - normalized_mse: 0.3661 - val_loss: 0.0367 - val_acc: 0.9559 -    val_normalized_mse: 0.2920

Another example:

Epoch 1/30
6912/6912 [==============================] - 28s 4ms/step - loss: 0.1251 -     acc: 0.8982 - normalized_mse: 0.5686 - val_loss: 0.1077 - val_acc: 0.9564 -     val_normalized_mse: 0.5302

The last one:

Epoch 1/30
6912/6912 [==============================] - 26s 4ms/step - loss: 0.1721 -    acc: 0.9400 - normalized_mse: 0.6751 - val_loss: 0.1582 - val_acc: 0.9582 -   val_normalized_mse: 0.6473

if I run my model for more than 30 epochs it starts overestimating. The size of the Input images is 256x256 each has quite a bit of features and it is grayscale - the output is binary image.

Can some one please help me to understand if I am doing something wrong? And how can I make my model more stable?

[SOLVED] For those who face the same problem just feed the kernel_initializer some random seed. Example kernel_initializer=he_normal(seed=seed_val). That will solve the issue.

Answer 1

Just a couple of ideas:

1. Batch size: 8 is quite a small batch, meaning the average loss that is computed might have high volatility. If you have enough memory, you could increase this.

2. Diversity of input: try adding batch normalisation layers in the encoder part, to try smoother the input for the conv layers. You said there are quite a few features, so perhaps this makes for noisy input, which would benefit from being normalised.

You could trying running the same experiment for 15 epochs, then plotting the training and validation losses (as they evolve perhaps, using the TensorBoard callback alongside your others). Do they follow any patterns or converge after some time?

You could try using different initilisation methods, or even gradient clipping, in order to make training a little smoother - constraining the size of the updates to weights during backpropagation.

Finally, another (brand-new!) result from research into GAN models, shows that progressively increasing the size of the inputs to your models might help to smooth learning and also extract a more robust set of features, which generalise better. Have a read of this section of an article from FastAI on their experience.

EDIT: information regarding the ELU activation

Using this activation my help learning, as it has a little more flexibility than e.g. the ELU, because it may also assume negative values. Here is an image from the original paper:

Here is the official definition (might slightly differ in the implementation of your framework):

The authors mentions that this activation assists in pushing the mean activation closer to zero, just as batch normalisation does. In your case this might mean simply getting past the bumpy initial epochs and converging a little more quickly.

The final major point that the authors highlight is that the models they trained also generalised better - so you might enjoy an improved performance with your model using ELUs versus a model using ReLUs (assuming both are trained for similar time).