Is applying dropout equivalent to zeroing output of random neurons in each mini-batch iteration and leaving rest of forward and backward steps in back-propagation unchanged? I'm implementing network from scratch in numpy.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Yes, although just to be super-duper-extra precise, bernoulli dropout is the same as zeroing-out random neurons (some people use other kinds of randomness and call it things like Gaussian dropout; see e.g. keras.io/api/layers/regularization_layers/gaussian_dropout) $\endgroup$– John MaddenNov 9, 2022 at 21:30
-
$\begingroup$ @Qbik please see edits to my reply below. $\endgroup$– hH1sG0n3Nov 10, 2022 at 9:07
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Indeed. To be precise, the dropout operation will randomly zero some of the input tensor elements with probability $p$, and furthermore the rest of the non-dropped out outputs are scaled by a factor of $\frac{1}{1-p}$ during training.
For example, see how elements of each tensor in the input (top tensor in output) are zeroed in the output tensor (bottom tensor in output) using pytorch.
m = nn.Dropout(p=0.5)
input = torch.randn(3, 4)
output = m(input)
print(input, '\n', output)
>>> tensor([[-0.9698, -0.9397, 1.0711, -1.4557],
>>> [-0.0249, -0.9614, -0.7848, -0.8345],
>>> [ 0.9420, 0.6565, 0.4437, -0.2312]])
>>> tensor([[-0.0000, -0.0000, 2.1423, -0.0000],
>>> [-0.0000, -0.0000, -1.5695, -1.6690],
>>> [ 0.0000, 0.0000, 0.0000, -0.0000]])
EDIT: please note the post has been updated to reflect Todd Sewell's addition in the comments.
-
1$\begingroup$ Note that non-dropped out elements are scaled by 1/(1-p) to compensate for the shift in average magnitude, so it's not just zeroing out some elements. $\endgroup$ Nov 9, 2022 at 20:12
-
$\begingroup$ That is very true, I omitted that info from the original pytorch docs for simplicity however in that way makes the post only half correct. Amended now to reflect your point. $\endgroup$– hH1sG0n3Nov 9, 2022 at 22:17