0
$\begingroup$

It is claimed Srivastava, Hinton, et al. that "dropout can be effectively applied in the hidden layers as well and that it can be interpreted as a form of model averaging" and that "training a neural network with dropout can be seen as training a collection of $2^n$ thinned networks with extensive weight sharing".

First: I can intuitively see how dropout is a form of model averaging. At every epoch during training, neuron values are stochastically set to 0. This is akin to having a different neural network (with different numbers of hidden neurons disabled) at every epoch. Is there a mathematical reason why this is interpreted as model averaging?

Second: where does the $2^n$ come from in the second quote?

$\endgroup$

1 Answer 1

0
$\begingroup$

Regarding first: Usually dropout is only active during the training phase of your model (Exceptions to this include models providing simple uncertainty quantification). Thus, during inference of new unseen inputs the whole model is used, essentially resulting in a superposition of all previously trained sub-models, which were created using dropout. I think this what they mean by averaging.

Regarding second: You need to consider the given context. Given a model with n neurons, which have the binary trait of being either enabled or disabled by dropout, results in 2^n possible sub-models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.