High, constant training loss with CNN

Question

I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:

x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)

lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]

sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

At human check I see that as a result, all predicted labels are a constant sequence.

What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).

Answer 1

I suspect you are facing a dying ReLU problem. Check the gradients for each layer and see if they are starting to become 0. Due to the way backpropagation works and a simple application of the chain rule, once a gradient is 0, it ceases to contribute to the model. Therefore, if you're model is stuck then it's likely that a significant number of your neurons are now dead.

Try using a LeakyRelu or Maxout and see if that helps.