0
$\begingroup$

I have 6 layer CNN model that (kind of) works with "GlobalAveragePooling2D()" in the end but instead if I flatten and add dense layers like below, it just does not learn. After 20 epochs loss and accuracy curves still doesn't change at all.

Do you think anything wrong the way I flatten or add dense layers in the end? I also printed out the shapes incase it helps.

model.add(Ativation('relu'))
model.add(BatchNormalization())
print("Conv6 after shape",model.output_shape)

model.add(Convolution2D(filters=64, kernel_size=(4,4),
                    strides=(2,2),
                    padding="same",
                    kernel_regularizer=l2(0.0001),
                    kernel_initializer="normal"))
model.add(BatchNormalization())                    
model.add(Activation('relu'))

print("before flat shape",model.output_shape)
model.add(Flatten())
model.add(Dropout(0.5))

print("after flat shape",model.output_shape)
model.add(Dense(200, activation='relu'))
print("after dense shape",model.output_shape)
model.add(Dense(2, activation='softmax'))

And here is output of full model:

0 shape (None, 1, 44100, 40)
1 shape (None, 1, 44100, 40)
2 shape (None, 1, 160, 40)
Conv3 after shape (None, 40, 160, 24)
Conv4 after shape (None, 40, 160, 24)
Conv5 after shape (None, 20, 80, 48)
Conv6 after shape (None, 10, 40, 48)
before flat shape (None, 5, 20, 64)
after flat shape (None, 6400)
after dense shape (None, 200)
$\endgroup$
3
  • 1
    $\begingroup$ Having dropout after convolution is rather unusual. Try to put it between the dense layers. $\endgroup$ Commented Jan 25, 2019 at 7:23
  • $\begingroup$ @Dmytro Prylipko It worked !:) can you create this as answer? $\endgroup$
    – Spring
    Commented Jan 25, 2019 at 22:21
  • $\begingroup$ Created the answer :) $\endgroup$ Commented Jan 26, 2019 at 9:01

1 Answer 1

3
$\begingroup$

Having dropout after convolution is rather unusual. Try to put it between the dense layers.

Explanation:

Dropout forces neuron to work with a random subset of the preceding neurons' activations. By doing so it breaks the co-adaptation of neurons and improves the generalization of the network. However, for convolutional layer co-adaptation is not an issue, as neighboring pixels of a feature map share their weights and guarantee to have different activations given different inputs in their receptive fields. It may only occur if two layers will learn the same kernel weights, but even then the problem is not so significant. Little number of weights in a convolution layer is a pledge of good generalization.

Here is a short article showing what happens when you use batchnorn/dropout between convolutions: Don’t Use Dropout in Convolutional Networks

$\endgroup$
1
  • $\begingroup$ Can you expand why it is 'unusual'? $\endgroup$ Commented Jan 26, 2019 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.