2
$\begingroup$

In the PyTorch nn module there are 2 types of dropouts:

  • A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.
  • A Dropout2d - Randomly zero out entire channels. Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.

My question is when we use CNN, which of the dropout we use in the Convolutional layers. In this paper it is being said we use the first kind. But I am not so sure what is the general practice.

$\endgroup$
0
$\begingroup$

Randomly zero out an entire channel represents, in my opinion, an information loss that is way too big. For that reason, I'd strongly recommend the first kind.

However, I would apply dropout only to the dense layers that follow the convolutional part. I think it's better to keep as much image data as possible. But this is just a personal preference/suggestion.

Hope this helps, otherwise let me know.

$\endgroup$
  • $\begingroup$ I wanted to know about the practice used when someone specifies dropout in a CNN $\endgroup$ – DuttaA Jun 6 at 10:25
0
$\begingroup$

Check this paper:

https://arxiv.org/pdf/1411.4280.pdf

In section 3.2 they discuss in detail why standard dropout fails in convolutional layers and the idea of spatial dropout. Fully connected networks learn via connections between single neurons while convolutions learn via features. In a sense, dropping out entire features instead of single neurons can be thought as the convolutional version of regular dropout.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.