For any layer in my neural net, should I apply dropout onto an entering vector, or on the pre-activated vector?

In other words:

$$\vec q=W\cdot \vec x$$ $$\vec h = activate(drop(\vec q))$$


$$\vec q=W\cdot (drop(\vec x)) $$ $$ \vec h = activate(\vec q)$$

I think the second variant is smoother (none of our current vector is fully dropped out, but is assembled from a mix of the dropped-out input) and is therefore softer.

  • $\begingroup$ I think that the result that you get is the same in both cases, i.e. h will be LOW (not HIGH), therefore the neuron will not "fire". This means that in both cases the contribution of the neuron in the training procedure will be the same, i.e. no contribution. I don't see why you assume that the 2nd variant would be smoother. $\endgroup$ – pcko1 Jun 2 '18 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.