5
$\begingroup$

In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example lstm_seq2seq.py which does the same thing but letter-by-letter uses only one LSTM in the encoder. My code looks like:

encoder = LSTM(latent_dim, return_sequences=True)(encoder_inputs)
encoder_outputs, state_h, state_c = LSTM(latent_dim, return_state=True)(encoder)

My question is why does the word-by-word version use two LSTM layers? And why is the return_sequences used?

$\endgroup$
3
$\begingroup$

About your first question: It is because word-by-word NLP model is more complicated than letter-by-letter one, so it needs a more complex network (more hidden units) to be modeled suitably.

About your second question: When you want to use two-staged LSTMs, the hidden sequence of first LSTM must be used as input of the second LSTM and the return_sequences option is used to do this.

$\endgroup$
  • $\begingroup$ Thanks @pythinker - I added some code to my question - so I've got the return_sequences and return_state around the right way? $\endgroup$ – Nic Cottrell Aug 12 '18 at 20:18
  • $\begingroup$ @NicCottrell Yes, that is right, but note that inclusion of return_state is optional. $\endgroup$ – pythinker Aug 12 '18 at 21:36
  • $\begingroup$ return_state=False by default though, so isn't it required for state_h, state_c to be returned? $\endgroup$ – Nic Cottrell Aug 13 '18 at 7:25
  • $\begingroup$ @NicCottrell It is absolutely required for state_h and state_c to be returned but not for encoder_outputs. $\endgroup$ – pythinker Aug 13 '18 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.