Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Does it make sense conceptually to apply dropout to an artificial neutral network while also applying bayesian regularization?
On one hand I would think that technically this should work just fine, but on the other hand if bayesian regularization is like finding the MAP estimate given some prior shouldn't the effect just be that it creates random noise in your outcomes?
It actually makes perfect sense to use both. Gal et al. provided a nice theory on how to interpret dropout through a Bayesian lense. In a nutshell, if you use dropout + regularization you are implicitly minimizing the same loss as for a Bayesian Neural Network (BNN), where you learn the posterior distribution over the network weights given the training data.
You can think of the fully Bayesian approach as taking it one step further on the road from MLE over MAP to Bayesian regression, as the former two provide only point estimates, while the latter gives access to the entire posterior distribution.
If you approximate a BNN with dropout, the introduced noise will take the role of drawing samples from the posterior distribution. The regularization will take the role of a prior, which is the same interpretation you would give it in the MAP-context. I don't think, however, that the represented priors are identical.
Either way, the prior helps to regulate the "spread" of the posterior. If you choose a large regularization parameter, which corresponds to a narrow prior distribution, the model will reduce variance in the posterior distribution as well.
If you only need a simple ANN with dropout and regularization, you probably don't care too much about using the approximate posterior after training, but I think this is still a nice perspective on things.
