Summarised from Deep Learning by Goodfellow Chapter 7, Page 262:

When we use Bagging models we average over all the predictions over the different models, which is written as $\frac{1}{k} \sum_{i=1}^kp^{(i)}(y|x)$ where $p^{(i)}$ is the probability of a prediction being true by the $i$ th model.

Building on this the book has written thus for dropout the expression would be: $\sum_{\mu}p(\mu)p(y|x,\mu)$ where $\mu$ is a certain configuration of mask or basically a certain sub-network. This makes sense.

Now, for even small NN's this is exponentially large. So what we do is instead of averaging input over all masks we average from 10-20 masks.. Also, the cost function is given by $E_{\mu}J(\theta, \mu)$.

My doubts are as follows:

  • Normally in dropout we do the "Inverted Dropout" as per Andre Ng which basically means we divide the output of a node by its probability of being propagated. After training is done by sampling each node's propagation probability, dividing the output by propagation probability we do the testing without any dropout and treating the Neural Network as a whole.

So the questions are:

  • How are these practical steps matching with the theory, the cost function in dropout is the normal cost function without any fancy probability term?
  • Where is the averaging being done over different sub-configurations?
  • $\begingroup$ So TL;DR please bridge the gap between theory and implementation, how the theory part is being taken care of in the implementation part? $\endgroup$ – DuttaA May 24 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.