Adding dense layer to CNN causes to stop learning

Question

I have 6 layer CNN model that (kind of) works with "GlobalAveragePooling2D()" in the end but instead if I flatten and add dense layers like below, it just does not learn. After 20 epochs loss and accuracy curves still doesn't change at all.

Do you think anything wrong the way I flatten or add dense layers in the end? I also printed out the shapes incase it helps.

model.add(Ativation('relu'))
model.add(BatchNormalization())
print("Conv6 after shape",model.output_shape)

model.add(Convolution2D(filters=64, kernel_size=(4,4),
                    strides=(2,2),
                    padding="same",
                    kernel_regularizer=l2(0.0001),
                    kernel_initializer="normal"))
model.add(BatchNormalization())                    
model.add(Activation('relu'))

print("before flat shape",model.output_shape)
model.add(Flatten())
model.add(Dropout(0.5))

print("after flat shape",model.output_shape)
model.add(Dense(200, activation='relu'))
print("after dense shape",model.output_shape)
model.add(Dense(2, activation='softmax'))

And here is output of full model:

0 shape (None, 1, 44100, 40)
1 shape (None, 1, 44100, 40)
2 shape (None, 1, 160, 40)
Conv3 after shape (None, 40, 160, 24)
Conv4 after shape (None, 40, 160, 24)
Conv5 after shape (None, 20, 80, 48)
Conv6 after shape (None, 10, 40, 48)
before flat shape (None, 5, 20, 64)
after flat shape (None, 6400)
after dense shape (None, 200)

Answer 1

Having dropout after convolution is rather unusual. Try to put it between the dense layers.

Explanation:

Dropout forces neuron to work with a random subset of the preceding neurons' activations. By doing so it breaks the co-adaptation of neurons and improves the generalization of the network. However, for convolutional layer co-adaptation is not an issue, as neighboring pixels of a feature map share their weights and guarantee to have different activations given different inputs in their receptive fields. It may only occur if two layers will learn the same kernel weights, but even then the problem is not so significant. Little number of weights in a convolution layer is a pledge of good generalization.

Here is a short article showing what happens when you use batchnorn/dropout between convolutions: Don’t Use Dropout in Convolutional Networks