3
$\begingroup$

I read about dropout and how it helps in catering overfitting. In simple layman terms, it randomly drops some of the neurons in forward propagation. My question is that since these neurons will be 0 in forward propagation, are we supposed to update them while backpropagating?

$\endgroup$

1 Answer 1

3
$\begingroup$

Correction made (see edit log):

No, there is no need for that.

For all dropped neuron, the corresponding activations are simply zeroed out so the corresponding gradient will also be zero.

Now because you do not want to update the parameters, you simply multiply the gradients with the same dropout mask as was used in your current iteration.

If you're using a library, you just have to do everything as it is usually done because this is already being taken care of.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.