Why my training and validation loss is not changing?

Question

I used MSE loss function, SGD optimization:

xtrain = data.reshape(21168, 21, 21, 21,1)    
inp = Input(shape=(21, 21, 21,1))
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(inp)
x = MaxPool3D(pool_size=(3, 3, 3),padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
encoded = Conv3D(filters=128, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)

print ("shape of decoded", K.int_shape(encoded))

x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(encoded)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = UpSampling3D((3, 3, 3))(x)

decoded = Conv3D(filters=1, kernel_size=(3, 3, 3), activation='relu', 
padding='same')(x)

print ("shape of decoded", K.int_shape(decoded))

autoencoder = Model(inp, decoded)
autoencoder.compile(optimizer='sgd', loss='mse')
autoencoder.fit(xtrain, xtrain,
                epochs=30,
                batch_size=32,
                shuffle=True,
                validation_split=0.2
                )  

Epoch 1/30
16934/16934 [==============================] - 446s - loss: 34552663732314849715                                                                                                                     15904.0000 - val_loss: 1893.9425
Epoch 2/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 3/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 4/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 5/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 6/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 7/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 8/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 9/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 10/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 11/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 12/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 13/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 14/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 15/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425

Answer 1

Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out most of the ReLU activations. This is known as the dying ReLU problem, although the issue here is not necessarily the choice of ReLU, you will probably get similar problems with other activations in the hidden layers.

You need to tone down some of the numbers that might be causing such a large initial loss, and maybe also make the weight updates smaller:

• Normalise your input data. The autoencoder is trying to match the input, and if the numbers are large here, this multiplies up to a large loss. If the input can have negative values (either naturally or due to the normalisation) then you should not have ReLU activation in the output layer otherwise it is not possible for the autoencoder to match the input and output values - in that case just have a linear output layer.

• Reduce the learning rate - in Keras SGD has default lr=0.01, try lower e.g. lr=0.0001. Also consider a more sophisticated optimiser than plain SGD, maybe Adam, Adagrad or RMSProp.

• Add some conservative weight initialisations. In Keras you can set the weight initialiser - see https://keras.io/initializers/ - however, the default glorot_uniform should already be OK in your case, so maybe you will not need to do this.

Answer 2

I want to also add an additional detail: check that you are not producing NaN or infinity to your model. This can happen very easily during minmax normalization where both min and max are equal to 0, or if the data just plainly includes inf/NaN.

This is especially nasty since normally if this happens on CPU your loss will simply read "NaN", alerting to you that something is wrong. However, if you are training on the GPU, this will not show up as NaN.