2
$\begingroup$

Can dropout be applied to convolution layers or just dense layers. If so, should it be used after pooling or before pooling and after applying activation?
Also I want to know whether batch normalization can be used in convolution layers or not.

I've seen here but I couldn't find valuable answers because of lacking reference.

$\endgroup$
2
$\begingroup$

In short, yes.

  • Batch Normalization Batch Normalization layer can be used in between two convolution layers, or between two dense layers, or even between a convolution and a dense layer. The important question is Does it help? Well, it is recommended to use BN layer as it shows improvement generally but the amount of improvement you will get is more problem dependent.
  • Dropout: Convolution layers, in general, are not prone to overfitting but it doesn't mean that you shouldn't use dropout. You can, but again this is problem dependent. For example, I was trying to build a network where I used Dropout in between conv blocks and my model got better with it. It is better if you apply dropout after pooling layer.
$\endgroup$
  • $\begingroup$ I have said the same Well, it is recommended to use BN layer as it shows improvement . Read the answer again $\endgroup$ – Nain Dec 16 '17 at 18:43
  • $\begingroup$ "Convolution layers, in general, are not prone to overfitting" - Just to offer another data point, I've certainly seen convolution layers overfit and have seen Dropout help considerably. Often in places where Batch Norm doesn't work! There are some math papers out there suggesting why, but I suspect a lot of people just try each one to see what works for their application. It depends on your manifold and that is often hard to understand. $\endgroup$ – Ezekiel Kruglick May 13 '18 at 4:06
2
$\begingroup$

So far this part hasn't been answered: "should it be used after pooling or before pooling and after applying activation?"

One team did some interesting experiments and found that, at least for residual networks, batch norm and activation should come before the weights layer (So it goes BN --> ReLU --> Weights). They found pooling placement could vary without detriment. If you try to replicate that paper note very carefully the paragraph on the exceptions in placement for the beginning and end of the residual stacks.

Publication: He et al. - 2016 - Identity mappings in deep residual networks

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer, would you mind explaining They found pooling placement could vary without detriment? $\endgroup$ – Media May 7 '18 at 1:42
  • 1
    $\begingroup$ @Media Sorry for any confusion, it just means they tried pooling before the convolutional layer and before the activation layer with no notable difference. Which makes numerical sense if you think about it. $\endgroup$ – Ezekiel Kruglick May 7 '18 at 23:47
1
$\begingroup$

Both Dropout and Batch Normalization can be used with convolutional layers; but it recommended to use BN and not Dropout (see links below).

Several tutorials apply BatchNormalization between Conv2D and Activation, before the MaxPooling2D

Like this:

model.add(Conv2D(64, 3, padding = "same"))
model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

BN may not speed up convergence; but it does (on average) improve generalization power (i.e. test accuracy).

See:

https://towardsdatascience.com/dont-use-dropout-in-convolutional-networks-81486c823c16

https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/

Here is the TMI version: https://kth.diva-portal.org/smash/get/diva2:955562/FULLTEXT01.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.