5
$\begingroup$

In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example lstm_seq2seq.py which does the same thing but letter-by-letter uses only one LSTM in the encoder. My code looks like:

encoder = LSTM(latent_dim, return_sequences=True)(encoder_inputs)
encoder_outputs, state_h, state_c = LSTM(latent_dim, return_state=True)(encoder)

My question is why does the word-by-word version use two LSTM layers? And why is the return_sequences used?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

About your first question: It is because word-by-word NLP model is more complicated than letter-by-letter one, so it needs a more complex network (more hidden units) to be modeled suitably.

About your second question: When you want to use two-staged LSTMs, the hidden sequence of first LSTM must be used as input of the second LSTM and the return_sequences option is used to do this.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks @pythinker - I added some code to my question - so I've got the return_sequences and return_state around the right way? $\endgroup$
    – Nic Cottrell
    Aug 12, 2018 at 20:18
  • $\begingroup$ @NicCottrell Yes, that is right, but note that inclusion of return_state is optional. $\endgroup$
    – pythinker
    Aug 12, 2018 at 21:36
  • $\begingroup$ return_state=False by default though, so isn't it required for state_h, state_c to be returned? $\endgroup$
    – Nic Cottrell
    Aug 13, 2018 at 7:25
  • $\begingroup$ @NicCottrell It is absolutely required for state_h and state_c to be returned but not for encoder_outputs. $\endgroup$
    – pythinker
    Aug 13, 2018 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.