1
$\begingroup$

I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:

x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)

lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]

sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

At human check I see that as a result, all predicted labels are a constant sequence.

What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).

| improve this question | | | | |
$\endgroup$
  • $\begingroup$ I don't use keras, but I've had this problem when my top-layer weights all get stuck at zero. I've noticed that this happens to me with a combination of dropout and large L2 penalties. Anyway it's worth checking the magnitudes of your top level weights. $\endgroup$ – generic_user Dec 15 '17 at 14:35
  • $\begingroup$ Too shallow network and too much of regularization. $\endgroup$ – enterML Dec 16 '17 at 18:03
1
$\begingroup$

I suspect you are facing a dying ReLU problem. Check the gradients for each layer and see if they are starting to become 0. Due to the way backpropagation works and a simple application of the chain rule, once a gradient is 0, it ceases to contribute to the model. Therefore, if you're model is stuck then it's likely that a significant number of your neurons are now dead.

Try using a LeakyRelu or Maxout and see if that helps.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.