Applying dropout effectively in CNN

Question

I am fairly new to deep learning and machine learning in general and have been trying to teach myself.

I’m interested in understanding when and how to effectively use dropout in a CNN. While researching, I’ve encountered conflicting advice regarding its placement. Some suggest using dropout after each fully-connected layer, while others recommend applying it only after every convolutional layer or even at both points. This post has an unaccepted answer which provides some brief details but I remain uncertain.

Placing it after every conv and fully connected layer seems overly aggressive. This is what I am currently doing. My CNN is trying to classify images using a custom dataset and I am struggling to get it past 50% validation accuracy so I am experiencing more with dropout:

def forward(self, x):
  x = self.bn1(F.relu(self.pool(self.conv1(x))))
  x = F.relu(self.conv11(x))
  x = F.relu(self.conv11(x))
  x = F.relu(self.conv11(x))

  x = self.bn2(F.relu(self.pool(self.conv2(x))))
  x = F.relu(self.conv21(x))
  x = F.relu(self.conv21(x))
  x = F.relu(self.conv21(x))
  x = self.bn3(F.relu(self.pool(self.conv3(x))))
  x = F.relu(self.conv31(x))
  x = F.relu(self.conv31(x))
  x = F.relu(self.conv31(x))
  x = self.bn4(F.relu(self.pool(self.conv4(x))))

  x = x.view(-1, 4 * 4 * 64)
  x = self.dropout(x)
  x = F.relu(self.fc(x))
  x = self.dropout(x)
  x = self.fc2(x)

  return torch.log_softmax(x, dim=1) 

I am wondering when exactly you would place dropout in the forward pass to ensure it's effective.

Would appreciate any insight. Thanks in advance.

Answer 1

Firstly, why do we use dropout in the first place? Dropout is a regularization technique designed to improve generalization and prevent overfitting.

With this in mind, you should not necessarily expect dropout to improve your validation accuracy. If you see a large divergence in train/val accuracy (indicative of overfitting), you should probably add more dropout. However, that may have the effect of lowering train accuracy without lowering validation accuracy. There is not necessarily a relationship between applying dropout and improving validation performance.

Secondly, consider the distinction between the location of dropout and the magnitude of dropout. Two scenarios:

1. Apply 90% dropout only in the model head
2. Apply 0.1% dropout after every layer

The first is likely to be more aggressive than the second, as it applies more dropout overall despite being applied in fewer places.

In terms of where to apply it, it depends on what model architecture you are going for.

Wide resnet tends to go bn/relu/conv/bn/relu/dropout/conv/residual. A more simple cnn might go conv/dropout/bn/relu. There are many configurations.

You can try add dropout wherever you see fit, at a low dropout percentage. See how that impact the model, train/val accuracy, and tune params.