Number of parameters in seq2seq model

Question

I am trying to compute the total number of parameters in Ilya Sutskever's seq2seq model. In the paper, the authors mention a total number of 384M trainable parameters, or to be exact:

The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM)

In short, their model has:

• one encoder LSTM and one decoder LSTM
• each LSTM has four layers
• each LSTM layer has 1000 units
• input: 1000-dimensional word embedding of 160K possible input words
• output: naive softmax over 80K possible words

Following this post and using the formula $$ num~params = 4 (nm + n^2) $$

I was able to compute:

For the encoder LSTM

• n = 1000 for all layers
• m = 1000 for the first hidden layer, because of 1000 dimensional embedding

• m = 1000 for all other layers, because that's the number of units in each hidden layer

  which yields:

• input -> first hidden layer: 8M

• first -> second : 8M
• second -> third : 8M
• third -> last : 8M
• total = 32M

which I am guessing is what they mean when they say 32M recurrent connections for the LSTM?

For the decoder LSTM

computations are exactly the same as above, with the caveat that the input size is 1000 because that's the output size of the encoder LSTM.

• total = 32M

Where are the other 320M parameters?

The computations made so far account for 64M parameters. We still haven't accounted for the final softmax layer, which admittedly could be pretty large since its output is 80K, and 320M can be suspiciously decomposed as 80K * 4K, but I'm not sure about this.

My questions are:

• can the naive softmax layer alone account for the 320M missing parameters?
• shouldn't we also be using the fact that this is a recurrent neural networks to compute additional weights, as in this post?

Thank you

Answer 1

I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation?

The reason for the remaining 320M parameters

• in the input of the encoder, computing the word embeddings requires taking a one-hot vector of 160K words and transforming it into a 1K dimensional word embedding. Thus this accounts for 160M parameters;
• the softmax layer of the decoder takes a 1K dimensional input (the hidden state of the last layer of LSTM cells) and transforms it into a vector of 80K probabilities. This accounts for 80M parameters;
• the input of the encoder is given not only by the hidden state of the decoder, but also by the previous output of the encoder. This requires taking a 80K dimensional vector of possible words and transforming it into a 1K dimensional word embedding. Thus this accounts for 80M parameters.

Does this explanation make sense?

Recurrent connections

As far as I understand, once the cell state of the LSTM is computed, it is transferred as is to the next timestep. Thus there are no additional parameters from the recurrent network (such as, e.g., was the case in this post).