0
$\begingroup$

Experimenting with the cifar10 dataset and faced with strange behavior when Dropout and BatchNorm don't help at all.

As I get:

  • Dropout - freezing some of the weights which helps us to prevent overfitting
  • BatchNorm - make training faster and more stable through normalization

Everything seems reasonable to use by default in NN. A course that I'm passing says the same. Experiments showed the opposite.

Experiments:

  1. Base NN (Conv2d and max pool) - showed the best result of growing accuracy and decreasing loss.
  2. Base NN + Droupout - 2nd place
  3. Base NN + Droupout + BatchNorm - 3th place

enter image description here

The question is why? Maybe I'm doing smt wrong? I'm kinda newbie in the Deep learning

NN itself:

   nn.Sequential(
        nn.Conv2d(3, 16, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),

        nn.Conv2d(16, 32, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),
        
        nn.Conv2d(32, 64, 3, padding=1),
        # nn.Dropout2d(0.2),
        # nn.BatchNorm2d(64),
        nn.ReLU(),
        nn.MaxPool2d(2,2, padding=1),
        
        nn.Flatten(),
        
        nn.Linear(1600, 500),
        nn.ReLU(),
        
        nn.Linear(500, n_classes)   
    )
$\endgroup$

2 Answers 2

1
$\begingroup$

Dropout is a regularization technique used to prevent overfitting by dropping some ratio of units of the neural network and on the other hand BatchNorm is used to normalize the units of each batch. The ordering is the problem. Why I do not have enough concrete explanation as to why placing dropout and batchNorm together is a problem, it is a common observation that the losses tend to go up when these two are placed side by side. It's generally a rule of thumb to make use of batchNorm instead of dropout in Convoluted layers and mostly before the activation function. Dropout on the other hand is preferably used in dense layers . There is some disharmony created when these two are placed together. So my best guess is to stick with batch Norm

$\endgroup$
3
  • $\begingroup$ Thank you for your answer! I made 4th experiment (Base NN + BatchNorm) and initially, it showed the best result but after the 5th epoch Base NN start winning again. I still don't get why we should use BatchNorm or Droupout if eventually results better without them? $\endgroup$
    – kirsanv43
    Commented Jul 12, 2023 at 16:57
  • $\begingroup$ Or does it depends and based on the stars' position in the sky we should decide whether to use BatchNorm or Droupout or not? $\endgroup$
    – kirsanv43
    Commented Jul 12, 2023 at 17:00
  • $\begingroup$ Not in all cases. BatchNorm is used to prevent something called internal covariance shift. Which is usually a problem that arises with datasets. Dropout is used to prevent overfitting. Cases when the training data is almost memorized by the model. You may not notice their effects yet untill you start working with more datasets. Also your graph shows training accuracy which is not a good way of showing the models real accuracy. Try with the validation set and see how your model performs. $\endgroup$
    – Chiho
    Commented Jul 13, 2023 at 9:09
1
$\begingroup$
  • Dropout is a regularization technique, used to prevent overfitting. It would not improve the accuracy if the network is shallow or of correct size for the dataset, it will rather rather hurt performance. I don't think your network needs dropout for cifar-10 dataset.
  • Also, you should try placing batch norm layer after activation layer. Since relu sets negative values to 0, the data being passed to the next layer will not be normalized.

    Read the answers here. A lot of good resources and explanations are shared.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.