# High, constant training loss with CNN

I try to solve a multi-character handwriting problem with CNN and I encounter with the problem that both training loss (~125.0) and validation loss (~130.0) are high and don't decrease. I use the following architecture with Keras:

x = Convolution2D(32, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, (3, 3), kernel_initializer='he_normal')(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
conv_out = (Dense(512, activation='relu', kernel_constraint=maxnorm(3)))(x)

lst = [x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13]

sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])


At human check I see that as a result, all predicted labels are a constant sequence.

What are the possible ways of improvement? The training data seems OK (at least as far as any pile of handwriting could be).

• I don't use keras, but I've had this problem when my top-layer weights all get stuck at zero. I've noticed that this happens to me with a combination of dropout and large L2 penalties. Anyway it's worth checking the magnitudes of your top level weights. – generic_user Dec 15 '17 at 14:35
• Too shallow network and too much of regularization. – enterML Dec 16 '17 at 18:03