I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras.

In Keras the input data structure for X is: (nb_samples, timesteps, input_dim).

Suppose that the shape of X is: (1000, timesteps = 10, 40).

1) Does this mean that the LSTM cells will only consider ‘batches’ of 10 previous time steps ?

2) Or is the output from LSTM cells passed between these sets of 10 timesteps I.e could you capture long term dependencies 50 timesteps out?

  • 1
    $\begingroup$ Great question. Have you considered posting this question to the Keras mailing list? It's quite an active list with both the founder of Keras and some very good "power users". I'm sure that they could answer your question in a fraction of the time it will take here. $\endgroup$ – I_Play_With_Data Feb 5 '19 at 21:44

The last number 40 has nothing to do with the sequence length, its a hyper-parameter setting that is basically the length of the vector 'representation' of each token in the sequence. In this case , its 40 length. If you set it to 40 and use input embeddings of input 300-dimension (common if Glove), then the 300-dimensional word gets mapped to a 40-dimensional word that goes through the LSTM permutations.

The idea is very similar to the number of 'kernel maps' in a CNN, if you're familiar with those. Its just a way to tell your network how many internal features you want your model to generate as your LSTM does its thing. More features means stronger representational power and better information flow, but at the possible cost of overfitting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.