3
$\begingroup$

The Neural Network language Model (NNLM) by Bengio et.al is a structure extensively used in machine translation, text summarization based on deep learning. What's the computational complexity of this model?

Knowing the complexity in terms of number of parameters helps in choosing the size of the training set and in determining what compute infrastructure is required.

$\endgroup$
1
$\begingroup$

NNLM has following sets of parameters ). Using $V$ to denote the number of words in the vocabulary:

  1. The matrix which creates the embedding for each word in the context . This distributed embedding is in $\mathbb{R}^{m}$ space. This matrix is $C :{V \times m}$
  2. The matrix which transforms the concatenated list of word embeddings (active in the current context of size $n-1$) to the hidden layer (of size $h$). This matrix is $H:{(n-1)m\times h}$
  3. The matrix which maps the hidden layer to unmormalized probabilities for each word in the vocabulary. This matrix is $V :{V \times h}$
  4. The matrix connecting the context word embeddings to the output layer. This connection is optional and has dimensions $W:{V \times (n-1)m}$ Note: while training using SGD, for a single example only $n-1$ words in the context are active out of words $V$ in the vocabulary. I have also omitted the parameter vectors for the bias terms $b:V \times 1$ when computing unnormalized outputs and when computing the hidden layer ($d: h \times 1$).

Hence my current understanding is that number of parameters in NNLM is: $$dim(C)+ dim(H)+ dim(V) + dim(W)= (n-1)m \times V + h \times (n-1)m+ V \times h + V \times (n-1)m$$

NNLM

$\endgroup$
1
  • $\begingroup$ I'm not quite sure about whether this answer is right (since I've answered my own question). Please correct it if it's wrong. $\endgroup$ – wabbit Mar 14 '16 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.