Correct order of operations involved into Dropout

Question

Suppose we have CNN with any hidden layer with activation followed by dropout layer. What is the correct precedence of activation and dropout operation if dropout implementation is inverted dropout and CNN mode is training mode? Do I need to compute activation in the first layer and then apply dropout with division by retain probability p, or I need to apply activation to the result of the division?

Say we have the following keras code

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

If I understand correctly, dropout with division by p will be applied to the activated result:

result = [survive_mask] * relu(output)/p

Is this correct? Wouldn't it be more natural to have

result = [survive_mask] * relu(output/p)

because otherwise dropout operation breaks activation value normalization (i.e. to [0, 1]) ?

Answer 1

The usual processing for your suggested layers:

model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

would be (reading left to right)

dense output -> relu -> apply dropout mask -> apply "inverse dropout" divide by p  

The precise combination may vary depending upon optimisations, and can in theory be changed a little without affecting the result (it doesn't matter to the end result numerically if we scale then mask or mask then scale for instance). However when dealing with vectorised optimisations (like those found in TensorFlow and Theano), it is normal to accept a percentage of "wasted" processing, and just have that naive left-to-right processing happen. It is often harder to parallelise decision branches than to simply process all items, even repeated multiplying by and adding zeroes for a significant fraction of each array.

There is no "normalization" implied by activation functions, so this is not a concern. From your comments, it seems you are worried that dividing by p could mean that the output of a neuron that would be between 0 and 1 (because you were using sigmoid activation for example) would now be between 0 and 1/p - i.e. larger. That is true. Is this a problem? No it is not, and in fact it is required. The impact of the larger values is fully compensated for by the weights learned in the connections between layers. If you used "vanilla" dropout then the weights would be correspondingly larger, but you would need to scale the outputs down (new range would be 0 to p) during testing/prediction.