1
$\begingroup$

I used MSE loss function, SGD optimization:

xtrain = data.reshape(21168, 21, 21, 21,1)    
inp = Input(shape=(21, 21, 21,1))
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(inp)
x = MaxPool3D(pool_size=(3, 3, 3),padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
encoded = Conv3D(filters=128, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)

print ("shape of decoded", K.int_shape(encoded))

x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(encoded)
x = Conv3D(filters=256, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = Conv3D(filters=512, kernel_size=(3, 3, 3), activation='relu',padding='same')(x)
x = UpSampling3D((3, 3, 3))(x)

decoded = Conv3D(filters=1, kernel_size=(3, 3, 3), activation='relu', 
padding='same')(x)

print ("shape of decoded", K.int_shape(decoded))

autoencoder = Model(inp, decoded)
autoencoder.compile(optimizer='sgd', loss='mse')
autoencoder.fit(xtrain, xtrain,
                epochs=30,
                batch_size=32,
                shuffle=True,
                validation_split=0.2
                )  

Epoch 1/30
16934/16934 [==============================] - 446s - loss: 34552663732314849715                                                                                                                     15904.0000 - val_loss: 1893.9425
Epoch 2/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 3/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 4/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 5/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 6/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 7/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 8/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 9/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 10/30
16934/16934 [==============================] - 444s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 11/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 12/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 13/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 14/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
Epoch 15/30
16934/16934 [==============================] - 445s - loss: 1896.7580 - val_loss                                                                                                                     : 1893.9425
$\endgroup$
  • $\begingroup$ I'm facing same problem but did get rid of it, how did you get rid of that? $\endgroup$ – gdmanandamohon Oct 27 '18 at 14:49
  • $\begingroup$ I suspect its happening because of having unsupported values like 'nan' or 'inf' etc $\endgroup$ – user63026 Nov 22 '18 at 2:05
5
$\begingroup$

Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out most of the ReLU activations. This is known as the dying ReLU problem, although the issue here is not necessarily the choice of ReLU, you will probably get similar problems with other activations in the hidden layers.

You need to tone down some of the numbers that might be causing such a large initial loss, and maybe also make the weight updates smaller:

  • Normalise your input data. The autoencoder is trying to match the input, and if the numbers are large here, this multiplies up to a large loss. If the input can have negative values (either naturally or due to the normalisation) then you should not have ReLU activation in the output layer otherwise it is not possible for the autoencoder to match the input and output values - in that case just have a linear output layer.

  • Reduce the learning rate - in Keras SGD has default lr=0.01, try lower e.g. lr=0.0001. Also consider a more sophisticated optimiser than plain SGD, maybe Adam, Adagrad or RMSProp.

  • Add some conservative weight initialisations. In Keras you can set the weight initialiser - see https://keras.io/initializers/ - however, the default glorot_uniform should already be OK in your case, so maybe you will not need to do this.

$\endgroup$
  • $\begingroup$ I tried with Adam also got the same problem without normalize .I want to give input without normalizing and i don't want to do any pre-processing. so is there any alternate to overcome with problem. I'll try with lr = 0.0001. $\endgroup$ – sp_713 Jun 9 '17 at 8:31
  • $\begingroup$ @sp_713 I haven't seen your data, but I suspect that normalising it, or at least scaling it into a smaller range per cell, could be required for numerical stability in training - which is why I put that option first. Why don't you want to do it? $\endgroup$ – Neil Slater Jun 9 '17 at 8:50

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.