5
$\begingroup$

Can dropout be applied to convolution layers or just dense layers. If so, should it be used after pooling or before pooling and after applying activation?
Also I want to know whether batch normalization can be used in convolution layers or not.

I've seen here but I couldn't find valuable answers because of lacking reference.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
5
$\begingroup$

In short, yes.

  • Batch Normalization Batch Normalization layer can be used in between two convolution layers, or between two dense layers, or even between a convolution and a dense layer. The important question is Does it help? Well, it is recommended to use BN layer as it shows improvement generally but the amount of improvement you will get is more problem dependent.
  • Dropout: Convolution layers, in general, are not prone to overfitting but it doesn't mean that you shouldn't use dropout. You can, but again this is problem dependent. For example, I was trying to build a network where I used Dropout in between conv blocks and my model got better with it. It is better if you apply dropout after pooling layer.
Improve this answer
$\endgroup$
2
  • $\begingroup$ I have said the same Well, it is recommended to use BN layer as it shows improvement . Read the answer again $\endgroup$
    – enterML
    Dec 16, 2017 at 18:43
  • 1
    $\begingroup$ "Convolution layers, in general, are not prone to overfitting" - Just to offer another data point, I've certainly seen convolution layers overfit and have seen Dropout help considerably. Often in places where Batch Norm doesn't work! There are some math papers out there suggesting why, but I suspect a lot of people just try each one to see what works for their application. It depends on your manifold and that is often hard to understand. $\endgroup$
    – Ezekiel Kruglick
    May 13, 2018 at 4:06
4
$\begingroup$

So far this part hasn't been answered: "should it be used after pooling or before pooling and after applying activation?"

One team did some interesting experiments and found that, at least for residual networks, batch norm and activation should come before the weights layer (So it goes BN --> ReLU --> Weights). They found pooling placement could vary without detriment. If you try to replicate that paper note very carefully the paragraph on the exceptions in placement for the beginning and end of the residual stacks.

Publication: He et al. - 2016 - Identity mappings in deep residual networks

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer, would you mind explaining They found pooling placement could vary without detriment? $\endgroup$
    – Green Falcon
    May 7, 2018 at 1:42
  • 2
    $\begingroup$ @Media Sorry for any confusion, it just means they tried pooling before the convolutional layer and before the activation layer with no notable difference. Which makes numerical sense if you think about it. $\endgroup$
    – Ezekiel Kruglick
    May 7, 2018 at 23:47
3
$\begingroup$

Both Dropout and Batch Normalization can be used with convolutional layers; but it recommended to use BN and not Dropout (see links below).

Several tutorials apply BatchNormalization between Conv2D and Activation, before the MaxPooling2D

Like this:

model.add(Conv2D(64, 3, padding = "same"))
model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

BN may not speed up convergence; but it does (on average) improve generalization power (i.e. test accuracy).

See this and this.

Here is the TMI version.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.