2
$\begingroup$

I am training an RNN on the following task: Given a sequence of thirty words, predict the next word.

Is there a benefit to having more than 30 cells (LSTM, GRU or plain RNN) in my network?
I've seen many examples online where similar networks are trained with multiple layers that each have 100 cells, but this does not make sense to me.
How does it help to have more cells than the length of the sequence? (in my case this length is 30)

I'm confused because from my understanding, each cell takes in two inputs
1. A new element of the sequence
2. The output from the previous cell
So after 30 cells, there will be no new sequence elements to input into the cell. Each cell will just be processing the output of the previous cell (receiving no new info).

I am using LSTM cells for this task (however, I'm guessing the actual type of RNN cell used is irrelevant).

$\endgroup$
1
$\begingroup$

You don't need to assign an RNN cell to each entry in a sequence, you just need one.

A single cell can process sequences because of its recursive nature(its output feeds back into itself):

$$ h_t = f(h_{t-1}, W, x_t) $$

where $h_t$ is the hidden state at step $t$, $W$ is the weight used for the cell, and $x_t$ is the current input at step $t$ .

Hopefully, you can see how it is only necessary for us one weight matrix, $W$, instead of multiple as your understanding would imply.

Where you got the notion of multiple RNN cells is probably when we feed the hidden state as the input, $x$, into another RNN cell. More layers of RNN cells, mean you can learn a more complex sequence.

Conclusively, your RNN cell cell can't work without new inputs, therefore it end when your sequence ends. Also, it is recommended to use LSTM or GRU over vanilla RNN units.

$\endgroup$
  • $\begingroup$ So what is being fed into each cell? Once the first cell has had the entire sequence fed into itself, it produces a single output, which per your description, is inputted into the next cell. But where does the second input come from? Or am I misunderstanding this and outputs at each point of the sequence are fed into the next cell. $\endgroup$ – James Dorfman Aug 28 '18 at 22:31
  • $\begingroup$ @JamesDorfman It gets fed into it a single instance of a sequence. E.g each word of a sentence would represent 'x' in the equation. Ill clarify the answer and edit x to xt to denote the input at step t. $\endgroup$ – Daniel Aug 28 '18 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.