3
$\begingroup$

I'm using a basic RNN as in the figure below (say for translation). The model has the following structure:

\begin{aligned} s_t &= \tanh(Ux_t + Ws_{t-1}) \\ o_t &= \mathrm{softmax}(Vs_t) \end{aligned}

  • Assume m is the vocabulary size and n that of the hidden layer.
  • If $x_{t}=\{0,1\}^{m}$ and U is a $ n \times m$ matrix then W is a $ n \times n $ matrix.
  • If $o_{t}$ is $\mathbb{R}^{k}$ and $s_{t}$ is $\mathbb{R}^{n}$then V is a $ k \times n $ matrix.

What's the # parameters for this RNN model?

a basic RNN

$\endgroup$
5
$\begingroup$

The entities W , U and V are shared by all steps of the RNN and these are the only parameters in the model described in the figure. Hence number of parameters to be learnt while training = $dim(W)+ dim(V)+ dim(U)$.

Based on data in the question this = $ n^{2}+ kn + nm$.

where,

  • n - dimension of hidden layer
  • k - dimension of output layer
  • m - dimension of input layer
| improve this answer | |
$\endgroup$
  • $\begingroup$ I'm not trying to gain reputation by answering my own question. Just wanted to document this somewhere since I found it useful and perhaps someone else will find it useful too. $\endgroup$ – wabbit Mar 8 '16 at 7:17
  • $\begingroup$ It is okay, as long as it is helpful to the users. Please add a more clear and detailed answer :) $\endgroup$ – Dawny33 Mar 8 '16 at 7:50
1
$\begingroup$

This is correct if one did not include biases. By including biases ($b_o$ and $b_h$). Number of parameters in $b_o$ is equal to number of outputs (k) and number of parameters in $b_h$ is equal to number of hidden layers (n). Hence the final value is:

$n^2 + n + mn + kn + k$

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.