I am trying to compute the total number of parameters in Ilya Sutskever's seq2seq model. In the paper, the authors mention a total number of 384M trainable parameters, or to be exact:
The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM)
In short, their model has:
- one encoder LSTM and one decoder LSTM
- each LSTM has four layers
- each LSTM layer has 1000 units
- input: 1000-dimensional word embedding of 160K possible input words
- output: naive softmax over 80K possible words
Following this post and using the formula $$ num~params = 4 (nm + n^2) $$
I was able to compute:
For the encoder LSTM
- n = 1000 for all layers
- m = 1000 for the first hidden layer, because of 1000 dimensional embedding
m = 1000 for all other layers, because that's the number of units in each hidden layer
input -> first hidden layer: 8M
- first -> second : 8M
- second -> third : 8M
- third -> last : 8M
- total = 32M
which I am guessing is what they mean when they say 32M recurrent connections for the LSTM?
For the decoder LSTM
computations are exactly the same as above, with the caveat that the input size is 1000 because that's the output size of the encoder LSTM.
- total = 32M
Where are the other 320M parameters?
The computations made so far account for 64M parameters. We still haven't accounted for the final softmax layer, which admittedly could be pretty large since its output is 80K, and 320M can be suspiciously decomposed as 80K * 4K, but I'm not sure about this.
My questions are:
- can the naive softmax layer alone account for the 320M missing parameters?
- shouldn't we also be using the fact that this is a recurrent neural networks to compute additional weights, as in this post?