Why should we use (or not) dropout on the input layer?

Question

People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it?

Adding dropout (given that it's randomized it will probably end up acting like another regularizer) should make the model more robust. It will make it more independent of a given set of features, which matter always, and let the NN find other patterns too, and then the model generalizes better even though we might be missing some important features, but that's randomly decided per epoch.

Is this an incorrect interpretation? What am I missing?

Isn't this equivalent to what we generally do by removing features one by one and then rebuilding the non-NN-based model to see the importance of it?

Answer 1

Why not, because the risks outweigh the benefits.

It might work in images, where loss of pixels / voxels could be somewhat "reconstructed" by other layers, also pixel/voxel loss is somewhat common in image processing. But if you use it on other problems like NLP or tabular data, dropping columns of data randomly won't improve performance and you will risk losing important information randomly. It's like running a lottery to throw away data and hope other layers can reconstruct the data.

In the case of NLP you might be throwing away important key words or in the case of tabular data, you might be throwing away data that cannot be replicated anyway else, like gens in a genome, numeric or factors in a table, etc.

I guess this could work if you are using an input-dropout-hidden layer model as you described as part of a larger ensemble though, so that the model focuses on other, less evident features of the data. However, in theory, this is already achieved by dropout after hidden layers.

Answer 2

It is not uncommon to use dropout on the inputs. In the original paper Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov - Dropout: A Simple Way to Prevent Neural Networks from Overfitting the authors usually use dropout with a retention rate of 50% for hidden units and 80% for (real-valued) inputs. For inputs that represent categorical values (e.g. one-hot encoded) a simple dropout procedure might not be appropriate.

They also argue that dropout applied to the inputs of Linear Regression yield a model that is similar to Ridge Regression where the dropout rate is related to the regularization strength [dropout adding variability/noise to the inputs leading to squeezing of the weights].

For deeper networks this is not quite as clear. but, in general, dropout adds noise to the data and is more useful for bigger datasets.

Approaches similar to dropout of inputs are also not uncommon in other algorithms, say Random Forests, where not all features need to be considered at every step using the same ideas.

The question is if adding dropout to the input layer adds a lot of benefit when you already use dropout for the hidden layers. In my experience, it doesn't for most problems. For some problems it makes more sense to inject noise earlier in the network to avoid overfitting from the beginning and sometimes only at later layers after some more complex features have already been built.