I have been reading the early paper on pre-training in NLP (https://arxiv.org/abs/1511.01432) and I can't understand what random word dropout means. The authors completely ignore explaining this method as if it was a standard thing. Can someone explain what they really do and what is the purpose of that?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Does this answer your question? Meaning of dropout $\endgroup$– Tom M.Commented Feb 24, 2020 at 22:04
-
1$\begingroup$ @Tom M. Not exactly. I know what a dropout is, but how do I apply it to words? Do I just randomly remove some of them (do I sample uniformly or based on frquency?) or maybe set them to some special token. If I have a sentence of length 5 words and 150 words then if I just remove 50% of words at random then effect maybe be different in those two cases. In a standard dropout size of the layer is the same for each train example $\endgroup$– WoofDoggyCommented Feb 24, 2020 at 22:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It is not uncommon that we can make sense of a sentence without reading it completely. Or when you are having a quick look at a document, you tend to oversee some words and still understand the main point. This is the intuition behind the word dropout.
Generally this is done by randomly dropping each word in a sequence following for example a Bernoulli distribution:
$X \leftarrow X \odot \vec{e}, \vec{e} ∼ B(n, p)$
where X is the index of the word token, n is the lenth of the sequence, and $\vec{e}$ is a vector with each word dropout state.
This is usually done after calculating the word embeddings, and the words selected to be left out are normally changed to the <UNK> equivalent embedding.
By doing this, we allow out model to learn more flexible ways of writing/convey meaning.
-
$\begingroup$ all right then, so I choose words at random from sequence and remove them. Yoav Goldberg in his NLP book says: "word dropout may also be beneficial for preventing overfitting and improving robustness by not letting the model rely too much on any single word being present". $\endgroup$ Commented Feb 25, 2020 at 11:38
-