1
$\begingroup$

I am looking to implement recurrent dropout (where recurrent connections between memory units of a recurrent layer such as LSTM/GRU/RNN are randomly set to 0) in Keras 2.3.1 on Theano backend on Python 3.6.

As of Keras 2.1.3, dropout for recurrent layers are no longer supported for the Theano backend:

Documenting this change further: the motivation for removing this feature in Theano is simply that although it would be technically possible to make it work, it would be hacky, i.e. it would reduce code readability, code maintainability, and importantly, it would be bug-prone. Since Theano development has been discontinued, we expect increasingly less Keras users to rely on the Theano backend, and thus the trade-off between supporting RNN dropout in Theano and having a nice and bug-free RNN codebase is tipping towards the latter.

Discussion

Unfortunately, I am unable to use a TensorFlow backend due to some server limitations nor roll back to an earlier version of Keras.

Any suggestions on how this may be implemented?

$\endgroup$
2
  • $\begingroup$ To implement dropout functionality look for building custom layer in keras that would help to build custom dropout layer. But still i would suggest try to move to tensorflow or downgrade keras. $\endgroup$ Jun 9 '20 at 19:56
  • $\begingroup$ Thanks Swapnil. I agree - especially since development efforts on Theano has been discontinued. For my use case unfortunately this doesn't seem to be possible at the moment, but a custom layer does seem to be the best approach. Also for anyone else looking for this, adding keras.layers.Dropout after a recurrent layer (rather than specifying the keyword arg) plays well with Theano for regular dropout. I may look into defining a custom layer that inherits from that to implement recurrent dropouts. $\endgroup$
    – Adam
    Jun 10 '20 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.