# How multi layer LSTM are interconnected?

I am trying to understand the layers in LSTM for my own implementation using Python. I started with Keras to getting familiarized with the layer flow.

I have tried the below code in Keras and I have the observations as follows

# LSTM MODEL
step_size = 3
model = Sequential()
model.add(LSTM(32, input_shape=(2, step_size), return_sequences = True))
model.add(LSTM(18))
model.add(Dense(1))
model.add(Activation('linear'))


And I got the below summary details for this implementation.

I tried to understand the internal structure of these layers and how these layer numbers ( 32, 18,1) mapped with matrix. I got the below matrix size from the weight matrix.

I can not see any relation between the output size and the weight matrix :(

Matrix Order

Layer 1

(3, 128)

(32, 128)

(128,)

Layer 2

(32, 72)

(18, 72)

(72,)

Layer 3

(18, 1)

(1,)


Now I am confused to find out the implementation logic inside the layer. Because I got three weight matrix in each LSTM layer and I can not connect the first LSTM(32) to the next LSTM(18) layer.

It will be thankful if any live implementation or flow diagram exists for this implementation. Thanks.

• What do you mean with connect? For what I can see they are fully connected, for instance the last layer has 19 parameters, that means 18 from the output of the right-layer and its bias. – Celius Stingher Aug 13 '19 at 21:35
• Connect means, Layer 1 output shape is (None,2,32) and Layer 2 input should also the same shape. What are the data to be added as input for layer 2. Is the input X[] and state matrix from layer 1? – felix Antony Aug 14 '19 at 18:27
• I investigated and found answer. Thank you – felix Antony Aug 16 '19 at 14:26