I'm a bit confused about dropout -- on one tutorial, it was described as basically an 'ensemble method' of sorts. This implies that you might need to create an ensemble of networks. Is this the case, where you would need to get a consensus from the ensemble at the end? Or is dropout run over one single network?
$\begingroup$
$\endgroup$
asked Sep 5, 2016 at 7:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Dropout is applied over one network. Sometimes (like with non-dropout networks) you will run your data through it multiple times before it converged and this number will be a bit higher on average with dropout but it is one network. Per layer you have a dropout probability during training and during testing/prediction you use the full network without dropout by multiplying the weights by (1-p) where p is the dropout probability. This is because during training (1-p) of the nodes are actually used.
Why this is related to ensembles is that every training instance is basically trained on a different network, by randomly dropping out nodes, forcing it to learn specific things using different nodes because it will not have all the nodes available at all times. It is not a traditional ensemble in that you combine multiple networks, just during training it acts a bit like it.
answered Sep 5, 2016 at 13:47
-
$\begingroup$ Ahhh.. So it's not that an instance is discarded at some node and is gone for good. Instances just skip certain nodes, all in one network only. $\endgroup$ Commented Sep 5, 2016 at 20:17
-
$\begingroup$ Exactly, this way all types of relevant information is captured by multiple nodes but in different ways, which enables somewhat of an ensemble-like behaviour. $\endgroup$ Commented Sep 5, 2016 at 20:24