12
$\begingroup$

I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.

$\endgroup$
22
$\begingroup$

Let's start with normal dropout, i.e. dropout only at training time. Here dropout serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly(e.g. multiply the dropout ratio). Such a model during test time can be understood as a average of an ensemble of neural networks.

Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.

For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same datapoint, your model could predict different values each time.

So the primary goal of Monte Carlo dropout is to generate random predictions and interpret them as samples from a probabilistic distribution. In the authors' words, they call it Bayesian interpretation.

Example: suppose you trained an dog/cat image classifier with Monte Carlo dropout. If you feed a same image to the classifier again and again, the classifier may be predicting dog 70% of the times while predicting cat 30% of the time. Therefore you can interpret the result in a probabilistic way: with 70% probability, this image shows a dog.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you +1 for putting it in simple and concise way. This went straight to my head in no time. $\endgroup$ – Haramoz Jan 16 '19 at 4:54
  • 1
    $\begingroup$ Why is MC dropout applied in training? In test I understand but why training? $\endgroup$ – Carlos Mougan May 26 at 21:12
  • $\begingroup$ Dropout is applied during training to allow the fully trained network to act as an ensemble of classifiers not just one network with one static configuration of weights. One can imagine that applying MC dropout on a network that didn't use dropout during training will force the network to perform predictions using configurations it's never optimised over and so the parameter estimations/outputs will essentially be garbage. It's worth noting here that dropout applied during training is not MC dropout, it's simply zeroing weights given some probabilistic function (usually Bernoulli I believe). $\endgroup$ – user8188120 May 29 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.