I have a question about types of RNN. Ian Goodfellow in his book Deep Learning writes:

Some examples of important design patterns for recurrent neural networks include the following:

• Recurrent networks that produce an output at each time step and have recurrent connections between hidden units, illustrated in figure 10.3.

• Recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step to the hidden units at the next time step, illustrated in figure 10.4

• Recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output, illustrated in figure 10.5.


Next I read about RNN article by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/ and his describe rnn architecture like relation model. enter image description here My question is: description of types RNN by Ian Goodfellow are equal to Andrej Karpathy? If not what it the diffrence beetween this descriptions?


1 Answer 1


They match up fairly well.

The first Goodfellow description is Karpathy's final "many to many" image. The output at each time step is based on the previous hidden state of the net and the input.

The third Goodfellow description directly corresponds to Karpathy's "many to one" image. This model reads an entire input sequence, and then produces one output.

The only difference is that the second description from Goodfellow's text isn't captured by Karpathy's image. Here's my rendition of what that description states.

  • $\begingroup$ Ok, thanks I wanted to be sure because I'm writing a master's thesis about it. $\endgroup$
    – lukassz
    Jun 20, 2018 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.