Recurrent neural network (LSTM) dimensions error

Question

I have data in a dataframe named ddf as follows:

labels          X
L1          [1,2,3,7,8,9...]
L1          [4,2,6,9,8,7...]
...
L2          [5,6,8,9,6,3...]
L2          [7,8,5,6,9,0...]
...

There are 250 rows, 7 labels and 2000 elements in every list under X. These 2000 elements are values of a signal over a period of about 60 seconds.

I am trying to build a recurrent neural network for above data. Following is my code:

Xall = ddf['X'].values
Xall = np.array(Xall)

ydf = pd.get_dummies(ddf.drop('X', axis=1))
Yall = np.array(ydf.values)

# Split the data
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(Xall, Yall,  test_size=0.1, random_state=0) 

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
model_lstm = Sequential()
model_lstm.add(Embedding(2000, 128))
model_lstm.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model_lstm.add(Dense(Yall.shape[1], activation='softmax'))
model_lstm.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model_lstm.fit(X_train, Y_train, epochs=50, verbose=True, validation_data=(X_test, Y_test))

However, I am getting error at second LSTM layer:

ValueError: Input 0 is incompatible with layer lstm_2: expected ndim=3, found ndim=2

I think this has something to do with LSTM arguments. Also are arguments of Embedding layer OK? How are both these adjusted? Where is the error coming from and how can it be solved? Thanks for your help.

Answer 1

EDIT after comments

Also are arguments of Embedding layer OK?

Yes. But You need to pass return_sequences = True in the first LSTM layer so that it will pass sequences to the next LSTM layer.

From the docs

return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

How are both these adjusted?

From the docs.

input_dim: Size of the vocabulary, i.e. maximum integer index + 1. This determines the largest integer in the input data. The largest integer in the input should be no larger than the vocabulary size. This should be the

output_dim: Dimension of the dense embedding. int >= 0

input_length: Length of input sequences, when it is constant. This argument is required if you are going to connect Flatten then Dense layers upstream (without it, the shape of the dense outputs cannot be computed).

I am posting code below for dummy data. The size of vocabulary has been taken 100.

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
input_array = np.random.randint(100, size=(250, 2000))
input_y = np.random.randint(7, size = (250))

Y_dumy = pd.get_dummies(input_y)
X_train, X_test, Y_train, Y_test = train_test_split(input_array, Y_dumy,  
test_size=0.1, random_state=0)

model = Sequential()
model.add(Embedding(input_dim = 100, output_dim = 64, input_length=2000))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences 
=True))
model.output_shape

#Output shape should be:
#model.output_shape = (None, 2000, 64)
#3D tensor with shape: (batch_size, sequence_length, output_dim)

model.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(Y_dumy.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics= 
['accuracy'])
model.fit(X_train, Y_train, epochs=50, verbose=True, validation_data= 
(X_test, Y_test))`

Where is the error coming from and how can it be solved?

I believe the error is coming because of absence of input_length. For similar errors please have a look at this post.

After the comments The error is coming because of return_sequences =False in the first LSTM layer.