Correct order of operations involved into Dropout

Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout and CNN mode is training mode? Do I need to compute activation in the first layer and then apply dropout with division by retain probability p, or I need to apply activation to the result of the division?

Say we have the following keras code

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))


If I understand correctly, dropout with division by p will be applied to the activated result:

result = [survive_mask] * relu(output)/p

Is this correct? Wouldn't it be more natural to have

result = [survive_mask] * relu(output/p)

because otherwise dropout operation breaks activation value normalization (i.e. to [0, 1]) ?

1 Answer

The usual processing for your suggested layers:

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))


would be (reading left to right)

dense output -> relu -> apply dropout mask -> apply "inverse dropout" divide by p


The precise combination may vary depending upon optimisations, and can in theory be changed a little without affecting the result (it doesn't matter to the end result numerically if we scale then mask or mask then scale for instance). However when dealing with vectorised optimisations (like those found in TensorFlow and Theano), it is normal to accept a percentage of "wasted" processing, and just have that naive left-to-right processing happen. It is often harder to parallelise decision branches than to simply process all items, even repeated multiplying by and adding zeroes for a significant fraction of each array.

There is no "normalization" implied by activation functions, so this is not a concern. From your comments, it seems you are worried that dividing by p could mean that the output of a neuron that would be between 0 and 1 (because you were using sigmoid activation for example) would now be between 0 and 1/p - i.e. larger. That is true. Is this a problem? No it is not, and in fact it is required. The impact of the larger values is fully compensated for by the weights learned in the connections between layers. If you used "vanilla" dropout then the weights would be correspondingly larger, but you would need to scale the outputs down (new range would be 0 to p) during testing/prediction.

• If you are not using batch normalization, then please explain your "activation value normalization" comment in more detail in the question, as I cannot see what else you could be referring to – Neil Slater Apr 5 '17 at 21:23
• well.. I just try to dive into very details (without any batching, just basics for now). "Activation value normalization" - here I meant that one of the purposes of activation is normalize of output, say, to [0, 1] in case of logistic activation. So, if we apply division by p after activation, output may be out of [0,1]. Is this ok? Doesn't this break the idea of activation as the-last-what-we-do with the input inside the layer? I understand that Dropout is the next layer, but it is pretty much 'fictive' layer though. – Serge P. Apr 5 '17 at 21:54
• @SergeP. The statement "one of the purposes of activation is normalize of output" is not true. It is the result if you use specific activations, such as sigmoid, but it does not happen with ReLU. Even if your activation was sigmoid though, it is not important here and the inverse dropout is doing the correct thing – Neil Slater Apr 5 '17 at 21:57
• This is what I wanted to hear. Thanks for detailed (as always) explanation, @NeilSlater! – Serge P. Apr 5 '17 at 22:04