3
$\begingroup$

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability $p$ and not present with a probability $1-p$. In the test phase, all units are present, but we multiply each of them with the probability.

Now, is it like, let we have 4 input units originally named a,b,c,d. In the training stage, after applying dropout, with a dropout rate of $0.5$, we are left with units a and c. So, As in the test stage, all the units are present, so, is it like, we multiply each of the units with $0.5$? Also, Is $p$ defined for each of the units in the network, or for the entire Neural Network?

Also, In doing so, how is the result same for training and test stage?

$\endgroup$
3
$\begingroup$

I guess you have not figured out the concept of dropout very well. First, the reason we apply it is that we add some noise to the architecture in order not be dependant on any special node. The reason is that it was observed that while training a network, after overfitting, the weights for some of neurons increases and cause the network to be dependant on them. By exploiting dropout, we are not dependant on any node anymore due to it is possible to drop it while training.

Now, answers to your question. First, you have to bear this point in mind that the probability shows the chance of dropping a node in a layer. Consequently, chance 0.5 does not mean you, for instance, will have those two nodes. It just means after employing dropout, the chance of dropping for each node is half. Dropout is used for layers. It is customary to use it in fully connected layers. You set the hyper-parameter and it is the chance of keeping the nodes in the layer. While testing, you don't drop any node. We don't multiply neurons to the probability. The probability specifies the chance of existence of that node.


Okey-doke! I update the answer. As you can read in the paper,

At test time, it is not feasible to explicitly average the predictions from exponentially many thinned models. However, a very simple approximate averaging method works well in practice. The idea is to use a single neural net at test time without dropout. The weights of this network are scaled-down versions of the trained weights. If a unit is retained with probability p during training, the outgoing weights of that unit are multiplied by p at test time as shown in Figure 2. This ensures that for any hidden unit the expected output (under the distribution used to drop units at training time) is the same as the actual output at test time. By doing this scaling, 2n networks with shared weights can be combined into a single neural network to be used at test time. We found that training a network with dropout and using this approximate averaging method at test time leads to signicantly lower generalization error on a wide variety of classication problems compared to training with other regularization methods.

I guess the easiest way to understand it is to watch this video. As you can see there are different implementations for that but the reason it is multiplied is that for any hidden unit the expected output (under the distribution used to drop units at training time) is the same as the actual output at test time. To be concise, it is done in order not to change the distribution that the outputs of the layer have.

$\endgroup$
  • $\begingroup$ If you look at the paper I referred to, in page 1931, the probability p is multiplied with weight w. So, pw. So, if a node has a probability of 0.3 for being dropped, will it be 0.3*w in test phase? $\endgroup$ – Akash Dubey Sep 27 '18 at 18:25
  • $\begingroup$ Not at all, it has a chance of dropping. I can't track the reference, is that the original dropout paper by Pr. Hinton? If it is not, it may be a different thing or it may be a kind of notation or something. $\endgroup$ – Media Sep 27 '18 at 18:27
  • $\begingroup$ Yes. It is the original paper published by Hinton. Did you look at it? $\endgroup$ – Akash Dubey Sep 27 '18 at 18:34
  • 1
    $\begingroup$ I got that. wait! I'll update. $\endgroup$ – Media Sep 27 '18 at 18:52
  • 1
    $\begingroup$ In my implementation, I used what in the video has been referred. You should know the reason it has been done. I guess it is better to consider the video due to being more up to date. It is more practical. $\endgroup$ – Media Sep 27 '18 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.