10
$\begingroup$

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed Dropout) he only put Dropout on the Dense layers, but that was because the hidden inner layers were convolutional.

Obviously, I can test for my specific model, but I wondered if there was a consensus on this?

$\endgroup$
  • 3
    $\begingroup$ some good discussion on dropout in recurrent networks in this paper if you're interested: arxiv.org/abs/1512.05287 Gal, Yarin, and Zoubin Ghahramani. "A theoretically grounded application of dropout in recurrent neural networks." Advances in neural information processing systems. 2016. $\endgroup$ – redhqs Sep 13 '18 at 15:57
  • 1
    $\begingroup$ Seems to confirm what @Media said below $\endgroup$ – BigBadMe Sep 13 '18 at 21:18
10
$\begingroup$

I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important thing about them is that they are not very well at memorising multiple things simultaneously. The logic of drop out is for adding noise to the neurons in order not to be dependent on any specific neuron. By adding drop out for LSTM cells, there is a chance for forgetting something that should not be forgotten. Consequently, like CNNs I always prefer to use drop out in dense layers after the LSTM layers.

$\endgroup$
  • 1
    $\begingroup$ I understand what you're saying, and it makes sense, but then, why does LSTM cell implementation in Keras or Tensorflow provide the ability to specify dropout (and recurrent dropout) if it will, in effect, undermine how an LSTM is supposed to function? $\endgroup$ – BigBadMe Sep 13 '18 at 16:28
  • 3
    $\begingroup$ In CNNs it is completely acceptable not to use them in conv layers due to the small number of weights in convolutional layers. In LSTMs on the other hand, the number of weights is not small. As I've mentioned in tasks that there are numerous things that have to be memorised, I try not to use dropout but it cases like the tense of verbs that you don't have many dependencies, I guess it is not very bad. By the way, it was my experience. There may be other answers for different application domains. $\endgroup$ – Media Sep 13 '18 at 17:01
  • 1
    $\begingroup$ Great Explaination by both answers!(+1) $\endgroup$ – Aditya Sep 13 '18 at 22:31
4
$\begingroup$

There is not a consensus that can be proved across all model types.

Thinking of dropout as a form of regularisation, how much of it to apply (and where), will inherently depend on the type and size of the dataset, as well as on the complexity of your built model (how big it is).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.