Better results when adding a dropout layer before a single layer classifier - counter intuitive result

Question

I am working on an multi-class image classification problem (with 9 classes), i am using a pretrained DenseNet121 (on ImageNet), i'm using Keras.

i am using densenet as a feature extractor, with a single layer as the classifier (with 9 neurons, one for each class).

base_model = DenseNet121(weights="ImageNet", include_avg=True, include_top=False)
x = Dropout(0.5)(base_model.output)
x = Dense(9, activation="softmax")(x)
model = Model(inputs=base_model.input, outputs=x)

By mistake i added a Dropout layer before the classification layer, as far as i understand, this means that, during training, if the model gives a correct prediction, but dropout takes effect, it might give a wrong prediction because the connection between the before layer neurons and the correct neuron is cut, so the model will be penalized, and vice-versa with a wrong prediction (if the next highest probability neuron is the correct class).

But what i got was an increase in accuracy by about 2.3% (from 94.49% to 96.82%), and decreased loss by about 0.02.

My question is, why did i get a better result with the dropout layer?

Thank you very much.

Answer 1

The general idea of Dropout is indeed to set the outputs of some layer to zero with a given probability. However, if you place the dropout layer before the classifier (i.e. after the base model), the dropout will apply to the inputs to the classifier layer (i.e. the outputs/features from the base model). As a result, the classification layer will only have access to a fraction of the features to make a prediction during training. In other words, the outputs of the classification layer remain untouched.

If you would actually put the dropout layer after the classification layer, performance should deteriorate as you would suspect. However, since you dropout the features, you just make the task for the classification layer harder. Typically, this leads to better generalisation performance (i.e. better validation/test set accuracy), which is what you observed.