Dropout and BatchNorm decrease speed of learning

Question

Experimenting with the cifar10 dataset and faced with strange behavior when Dropout and BatchNorm don't help at all.

As I get:

• Dropout - freezing some of the weights which helps us to prevent overfitting
• BatchNorm - make training faster and more stable through normalization

Everything seems reasonable to use by default in NN. A course that I'm passing says the same. Experiments showed the opposite.

Experiments:

1. Base NN (Conv2d and max pool) - showed the best result of growing accuracy and decreasing loss.
2. Base NN + Droupout - 2nd place
3. Base NN + Droupout + BatchNorm - 3th place

The question is why? Maybe I'm doing smt wrong? I'm kinda newbie in the Deep learning

NN itself:

   nn.Sequential(
        nn.Conv2d(3, 16, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),

        nn.Conv2d(16, 32, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),
        
        nn.Conv2d(32, 64, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(64),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),
        
        nn.Flatten(),
        
        nn.Linear(1600, 500),
        nn.ReLU(),
        
        nn.Linear(500, n_classes)   
    )

Answer 1

Dropout is a regularization technique used to prevent overfitting by dropping some ratio of units of the neural network and on the other hand BatchNorm is used to normalize the units of each batch. The ordering is the problem. Why I do not have enough concrete explanation as to why placing dropout and batchNorm together is a problem, it is a common observation that the losses tend to go up when these two are placed side by side. It's generally a rule of thumb to make use of batchNorm instead of dropout in Convoluted layers and mostly before the activation function. Dropout on the other hand is preferably used in dense layers . There is some disharmony created when these two are placed together. So my best guess is to stick with batch Norm

Answer 2

• Dropout is a regularization technique, used to prevent overfitting. It would not improve the accuracy if the network is shallow or of correct size for the dataset, it will rather rather hurt performance. I don't think your network needs dropout for cifar-10 dataset.
• Also, you should try placing batch norm layer after activation layer. Since relu sets negative values to 0, the data being passed to the next layer will not be normalized.
  
  Read the answers here. A lot of good resources and explanations are shared.