Hinton advocates tuning dropout in conjunction with tuning the size of your hidden layer. Increase your hidden layer size(s) with dropout turned off until you perfectly fit your data. Then, using the same hidden layer size, train with dropout turned on. This should be a nearly optimal configuration
A good initial configuration for the hidden layers is 50%. If applying dropout to an input layer, it's best to not exceed 25%.
By intuition I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers
If I were to try to generalize, I'd say that it's all about balancing an increase in the number of parameters of your network without over-fitting. So if e.g. you start with a reasonable architecture and amount of dropout, and you want to try increasing the number of neurons in some layer, you will likely want to increase the dropout proportionately, so that the same number of neurons are not-dropped. If p=0.5
was optimal for 100
neurons, a good first guess for 200
neurons would be to try p=0.75
.
Also, I get the feeling that dropout near the top of the network can be more damaging than dropout near the bottom of the network, because it prevents all downstream layers from accessing that information.
As to why 0.5
is generally used, I think its because tuning things like the dropout parameter is really something to be done when all of the big choices about architecture have been settled. It might turn out that 0.10
is better, but it takes a lot of work to discover that, and if you change the filter size of a conv-net or change the overall number of layers or do things like that, you're going to have to re-optimize that value.
So 0.5
is seen as a sort of placeholder value until we are at the stage where we are chasing down percentage points or fractions of a percentage point.
Original Paper and the opening paragraph is from the original paper...