I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
Let's start with normal dropout, i.e. dropout only at training time. Here, dropout serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly (e.g. multiplied by the keep ratio, which is 1 - dropout_ratio). Such a model during test time can be understood as an average of an ensemble of neural networks.
Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.
For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same data point, your model could predict different values each time.
So, the primary goal of Monte Carlo dropout is to generate random predictions and interpret them as samples from a probabilistic distribution. In the authors' words, they call it Bayesian interpretation.
Example: suppose you trained a dog/cat image classifier with Monte Carlo dropout. If you feed a same image to the classifier again and again, the classifier may be predicting dog 70% of the times while predicting cat 30% of the time. Therefore, you can interpret the result in a probabilistic way: with 70% probability, this image shows a dog.
-
$\begingroup$ Thank you +1 for putting it in simple and concise way. This went straight to my head in no time. $\endgroup$ Jan 16, 2019 at 4:54
-
2$\begingroup$ Why is MC dropout applied in training? In test I understand but why training? $\endgroup$ May 26, 2020 at 21:12
-
2$\begingroup$ Dropout is applied during training to allow the fully trained network to act as an ensemble of classifiers not just one network with one static configuration of weights. One can imagine that applying MC dropout on a network that didn't use dropout during training will force the network to perform predictions using configurations it's never optimised over and so the parameter estimations/outputs will essentially be garbage. It's worth noting here that dropout applied during training is not MC dropout, it's simply zeroing weights given some probabilistic function (usually Bernoulli I believe). $\endgroup$ May 29, 2020 at 9:45