LSTM Long Term Dependencies Keras

Question

I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras.

In Keras the input data structure for X is: (nb_samples, timesteps, input_dim).

Suppose that the shape of X is: (1000, timesteps = 10, 40).

1) Does this mean that the LSTM cells will only consider ‘batches’ of 10 previous time steps ?

2) Or is the output from LSTM cells passed between these sets of 10 timesteps I.e could you capture long term dependencies 50 timesteps out?

Answer 1

The last number 40 has nothing to do with the sequence length, its a hyper-parameter setting that is basically the length of the vector 'representation' of each token in the sequence. In this case , its 40 length. If you set it to 40 and use input embeddings of input 300-dimension (common if Glove), then the 300-dimensional word gets mapped to a 40-dimensional word that goes through the LSTM permutations.

The idea is very similar to the number of 'kernel maps' in a CNN, if you're familiar with those. Its just a way to tell your network how many internal features you want your model to generate as your LSTM does its thing. More features means stronger representational power and better information flow, but at the possible cost of overfitting.