Understanding input of LSTM

Question

I am a little confused with the input of LSTM. Basicaly my train input data is of shape (53394, 3). I reshaped my 2D data into 3D data in order to set it according to the input of LSTM. I have two configurations:

1. trainX = (53394, 3, 1)
   model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))
   

2. trainX = (53394,1, 3)
   model.add(LSTM(32,input_shape =(trainX.shape[1], trainX.shape[2]))
   

I want to understand how input is feed into the LSTM in both the cases. like one neuron is taking one column value or one complete row with three columns is going as input to one neuron in input layer.

Answer 1

LSTM layers work on 3D data with the following structure (nb_sequence, nb_timestep, nb_feature).

• nb_sequence corresponds to the total number of sequences in your dataset (or to the batch size if you are using mini-batch learning).
• nb_timestep corresponds to the size of your sequences.
• nb_feature corresponds to number of features describing each of your timesteps.

Thus an LSTM layer will work like this :
Let $t_i$ be the $i^{th}$ timestep of sequence $seq_j$, with $i \in [0, nb\_timestep], \ j \in [0, nb\_sequence]$.
An LSTM layer will make a prediction $p_i$ according to the $nb\_feature$ descriptors of $t_i$ with respect to its hidden state which is a representation of the timesteps $t_0$ to $t_{i-1}$.

---

Now, let's see what this means for your two configurations. For the sake of the explanation, I will suppose that we have sequences of words

• For (53394, 3, 1) the LSTM will work on 53394 different sequences. Each sequence is 3 words long and each word is described through one feature only. For the first word of each sequence, the LSTM will make a prediction on its sole descriptor. For the second word, the prediction will be done from the unique descriptor with respect to what the first word was. Finally, for the third and last word in the sequence, the LSTM will emit a prediction from the unique descriptor with respect to what the two previous words were. Then, the LSTM begins the process anew for the following sequence.
• For (53394, 1, 3), your sequences contain only one word which is described through 3 features. Sequences of one word are not really sequences, so the LSTM layer will not be useful in this case.

Hope it clears up how data are fed to an LSTM !

---

NB. Not related to the question but it may help : from your original shape, it seems your dataset contains 53394 words described with 3 features. If I am right, you would need a 3D shape like (53394, nb_timestep, 3) but with $nb\_timestep \neq 1 $. What you need then, is to define some window instead of reshaping your data.

Answer 2

Input of Recurrent cells (LSTM but also GRU and basic RNN cells) follows this pattern:

( number of observations , lenght of input sequence , number of variables )

Assuming your lenght of input sequence is 3, and only one variable, you can go with:

LSTM(32, input_shape=(3, 1))

As you can see, when you declare an LSTM() layer you don't need to specify the number of observations, Keras is taking care of that automatically.

Before that, you have to reshape your matrix to (53394, 3, 1). You can use np.expand_dims() or the .reshape() command.