2
$\begingroup$

I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, where you would need to get a consensus from the ensemble at the end? Or is dropout run over one single network?

$\endgroup$
4
$\begingroup$

Dropout is applied over one network. Sometimes (like with non-dropout networks) you will run your data through it multiple times before it converged and this number will be a bit higher on average with dropout but it is one network. Per layer you have a dropout probability during training and during testing/prediction you use the full network without dropout by multiplying the weights by (1-p) where p is the dropout probability. This is because during training (1-p) of the nodes are actually used.

Why this is related to ensembles is that every training instance is basically trained on a different network, by randomly dropping out nodes, forcing it to learn specific things using different nodes because it will not have all the nodes available at all times. It is not a traditional ensemble in that you combine multiple networks, just during training it acts a bit like it.

$\endgroup$
  • $\begingroup$ Ahhh.. So it's not that an instance is discarded at some node and is gone for good. Instances just skip certain nodes, all in one network only. $\endgroup$ – Monica Heddneck Sep 5 '16 at 20:17
  • $\begingroup$ Exactly, this way all types of relevant information is captured by multiple nodes but in different ways, which enables somewhat of an ensemble-like behaviour. $\endgroup$ – Jan van der Vegt Sep 5 '16 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.