Dropout in a CNN vs Dropout in a FCNN

Question

In the PyTorch nn module there are 2 types of dropouts:

• A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.

• A Dropout2d - Randomly zero out entire channels. Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.

  • From Documentation

My question is when we use CNN, which of the dropout we use in the Convolutional layers. In this paper it is being said we use the first kind. But I am not so sure what is the general practice.

Answer 1

Check this paper:

https://arxiv.org/pdf/1411.4280.pdf

In section 3.2 they discuss in detail why standard dropout fails in convolutional layers and the idea of spatial dropout. Fully connected networks learn via connections between single neurons while convolutions learn via features. In a sense, dropping out entire features instead of single neurons can be thought as the convolutional version of regular dropout.

Answer 2

Randomly zero out an entire channel represents, in my opinion, an information loss that is way too big. For that reason, I'd strongly recommend the first kind.

However, I would apply dropout only to the dense layers that follow the convolutional part. I think it's better to keep as much image data as possible. But this is just a personal preference/suggestion.

Hope this helps, otherwise let me know.