Probability of dropout growth

Question

In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer?

By intuition, I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers. For example, passing from a p_keep = 0.8 to p_keep=0.5, especially when the inputs features are all relevant. Where p_keep is the probability of keeping a node.

For instance:

Layer1: p_keep=0.9
Layer2: p_keep=0.7
Layer3: p_keep=0.6
Layer4: p_keep=0.55
Layer5-n: p_keep=0.5

Answer 1

Hinton advocates tuning dropout in conjunction with tuning the size of your hidden layer. Increase your hidden layer size(s) with dropout turned off until you perfectly fit your data. Then, using the same hidden layer size, train with dropout turned on. This should be a nearly optimal configuration

A good initial configuration for the hidden layers is 50%. If applying dropout to an input layer, it's best to not exceed 25%.

By intuition I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers

If I were to try to generalize, I'd say that it's all about balancing an increase in the number of parameters of your network without over-fitting. So if e.g. you start with a reasonable architecture and amount of dropout, and you want to try increasing the number of neurons in some layer, you will likely want to increase the dropout proportionately, so that the same number of neurons are not-dropped. If p=0.5 was optimal for 100 neurons, a good first guess for 200 neurons would be to try p=0.75.

Also, I get the feeling that dropout near the top of the network can be more damaging than dropout near the bottom of the network, because it prevents all downstream layers from accessing that information.

As to why 0.5 is generally used, I think its because tuning things like the dropout parameter is really something to be done when all of the big choices about architecture have been settled. It might turn out that 0.10 is better, but it takes a lot of work to discover that, and if you change the filter size of a conv-net or change the overall number of layers or do things like that, you're going to have to re-optimize that value. So 0.5 is seen as a sort of placeholder value until we are at the stage where we are chasing down percentage points or fractions of a percentage point.

Original Paper and the opening paragraph is from the original paper...