Connection between Embedding and LSTM and Dense layer

Question

I am building a "predict next word" model using the following model architecture. The codes fine, but I have a few questions:

# define model
model = Sequential()
model.add(Embedding(vocab_size, 50, input_length=seq_length))
model.add(LSTM(100, return_sequences=True))
model.add(LSTM(100))
model.add(Dense(100, activation='relu'))
model.add(Dense(vocab_size, activation='softmax'))
print(model.summary())

Between the Embedding and LSTM layers, are there 50 connection weights, or 50 * 100 connection weights? Based on the documentation, this LSTM block contains 100 cells. So I am not sure how exactly does the Embedding layer connect to the LSTM layer?

Also, between LSTM(100) layer and the Dense(100, activation='relu') layer, are there 100 connection weights or 100 * 100 connection weights?

Thanks!

Answer 1

The embedding layer has an output shape of 50. The first LSTM layer has an output shape of 100. How many parameters are here? Take a look at this blog to understand different components of an LSTM layer.

Then you can get the number of parameters of an LSTM layer from the equations or from this post.

Now, between LSTM(100) layer and the Dense(100, activation='relu') layer, there should be 100*(100 + 1) parameters. The additional 1 is for the bias.