1
$\begingroup$

The final step in dropout regularization is to multiply the weights by the dropout probability. This is motivated by analogy to bagging: averaging the weights of multiple nets. But it isn't truly that because of Jensen's inequality.

The effect of the averaging is basically just to shrink the weights by some factor. Why average at all? Has anyone tried simply not averaging, and controlling the magnitude of the parameters with e.g. L2 regularization? One would still get the benefits of preventing codependence, but one would need to regularize some other way.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

With dropout, the sum of activations feeding each neuron during training and testing needs to be roughly the same. The problem is that during training some neurons are masked out, and this does not happen during testing. Therefore there has to be a scaling difference between train and test phases using dropout, otherwise each neuron would be receiving larger magnitude inputs and the network as a whole could behave very differently.

This is not mathematically the same as averaging the weights - although the effect is compared intuitively to bagging many related neural networks and averaging their output. Instead, it is usually explained that dropout scales the neuron activations to have same expected total magnitude when dropout is active (during training) vs inactive (during testing/prediction).

Applying L2 or any other kind of parameter-controlling regularisation technique will not help with this, because these do not change any of the activations between train and test phases.

You either treat training activation values as canonical and adjust activations to be lower at test time (multiply by keep probability $p$), which is "classic dropout" or you treat test activation values as canonical and multiply activations by $\frac{1}{p}$ during training, which is called "inverse dropout". The latter approach is nowadays more common, as it is possible to completely ignore dropout layers during prediction that way.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the answer. I hadn't heard of inverse dropout. I suppose that one would multiply one's update by $1/p$ after each backward pass? $\endgroup$
    – generic_user
    Dec 1, 2017 at 15:15
  • $\begingroup$ But still there is something I don't get about your answer: a neuron only "sees" a weighted sum of activations during the forward pass. Testing/prediction obviously only involves the forward pass. But dropout (at least as I've understood and implemented it) is done on the backward pass only. Right? $\endgroup$
    – generic_user
    Dec 1, 2017 at 15:17
  • $\begingroup$ @generic_user: Dropout is done on both forward and backward pass, and the same neurons must be dropped out consistently for each example (each example gets a different set of dropped out neurons, but the forward and backward passes for that example must have the same activation masks). If you have implemented only on the backward pass, the training may still sort of work, just your gradients become noisy and you don't really get much benefit. $\endgroup$
    – Neil Slater
    Dec 1, 2017 at 15:58
  • $\begingroup$ "I suppose that one would multiply one's update by 1/p1/p after each backward pass?" - you don't multiply the backprop gradients or weight updates like that in dropout . . . you multiply the activations during forward pass (either during training or during test depending on which variant you have implemented) $\endgroup$
    – Neil Slater
    Dec 1, 2017 at 16:01
  • $\begingroup$ Hmm, maybe I need to go back and re-read some things. I had been viewing dropout as analogous to the method of random subspaces as implemented in RF. It actually works fairly well as I've implemented it, but who knows how much I might still be able to improve it implementing it more canonically. $\endgroup$
    – generic_user
    Dec 1, 2017 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.