I am having difficulties in understanding what is Monte Carlo dropout and what is the goal of using it instead of normal dropout. How to enable dropout at inference time in Keras is understood. But can anyone experienced explain is simple words? Thanks :)
Let's start with normal dropout, i.e. dropout only at training time. Here dropout serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly(e.g. multiply the dropout ratio). Such a model during test time can be understood as a average of an ensemble of neural networks.
Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.
For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same datapoint, your model could predict different values each time.
So the primary goal of Monte Carlo dropout is to generate random predictions and interpret them as samples from a probabilistic distribution. In the authors' words, they call it Bayesian interpretation.
Example: suppose you trained an dog/cat image classifier with Monte Carlo dropout. If you feed a same image to the classifier again and again, the classifier may be predicting dog 70% of the times while predicting cat 30% of the time. Therefore you can interpret the result in a probabilistic way: with 70% probability, this image shows a dog.