# Why my training and validation loss is not changing?

I used MSE loss function, SGD optimization:

xtrain = data.reshape(21168, 21, 21, 21,1)
inp = Input(shape=(21, 21, 21,1))
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(inp)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
encoded = Conv3D(filters=128, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)

print ("shape of decoded", K.int_shape(encoded))

x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(encoded)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = UpSampling3D((3, 3, 3))(x)

decoded = Conv3D(filters=1, kernel_size=(3, 3, 3), activation='relu',

print ("shape of decoded", K.int_shape(decoded))

autoencoder = Model(inp, decoded)
autoencoder.compile(optimizer='sgd', loss='mse')
autoencoder.fit(xtrain, xtrain,
epochs=30,
batch_size=32,
shuffle=True,
validation_split=0.2
)

Epoch 1/30
16934/16934 [==============================] - 446s - loss: 34552663732314849715                                                                                                                     15904.0000 - val_loss: 1893.9425
Epoch 2/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 3/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 4/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 5/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 6/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 7/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 8/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 9/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 10/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 11/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 12/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 13/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 14/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 15/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425

• I'm facing same problem but did get rid of it, how did you get rid of that? Oct 27 '18 at 14:49
• I suspect its happening because of having unsupported values like 'nan' or 'inf' etc Nov 22 '18 at 2:05

Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out most of the ReLU activations. This is known as the dying ReLU problem, although the issue here is not necessarily the choice of ReLU, you will probably get similar problems with other activations in the hidden layers.

You need to tone down some of the numbers that might be causing such a large initial loss, and maybe also make the weight updates smaller:

• Normalise your input data. The autoencoder is trying to match the input, and if the numbers are large here, this multiplies up to a large loss. If the input can have negative values (either naturally or due to the normalisation) then you should not have ReLU activation in the output layer otherwise it is not possible for the autoencoder to match the input and output values - in that case just have a linear output layer.

• Reduce the learning rate - in Keras SGD has default lr=0.01, try lower e.g. lr=0.0001. Also consider a more sophisticated optimiser than plain SGD, maybe Adam, Adagrad or RMSProp.

• Add some conservative weight initialisations. In Keras you can set the weight initialiser - see https://keras.io/initializers/ - however, the default glorot_uniform should already be OK in your case, so maybe you will not need to do this.

• I tried with Adam also got the same problem without normalize .I want to give input without normalizing and i don't want to do any pre-processing. so is there any alternate to overcome with problem. I'll try with lr = 0.0001. Jun 9 '17 at 8:31
• @sp_713 I haven't seen your data, but I suspect that normalising it, or at least scaling it into a smaller range per cell, could be required for numerical stability in training - which is why I put that option first. Why don't you want to do it? Jun 9 '17 at 8:50

I want to also add an additional detail: check that you are not producing NaN or infinity to your model. This can happen very easily during minmax normalization where both min and max are equal to 0, or if the data just plainly includes inf/NaN.

This is especially nasty since normally if this happens on CPU your loss will simply read "NaN", alerting to you that something is wrong. However, if you are training on the GPU, this will not show up as NaN.