Types of Recurrent Neural Networks

Question

I have a question about types of RNN. Ian Goodfellow in his book Deep Learning writes:

Some examples of important design patterns for recurrent neural networks include the following:

• Recurrent networks that produce an output at each time step and have recurrent connections between hidden units, illustrated in figure 10.3.

• Recurrent networks that produce an output at each time step and have recurrent connections only from the output at one time step to the hidden units at the next time step, illustrated in figure 10.4

• Recurrent networks with recurrent connections between hidden units, that read an entire sequence and then produce a single output, illustrated in figure 10.5.

CHAPTER 10. SEQUENCE MODELING: RECURRENT AND RECURSIVE NETS

Next I read about RNN article by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/ and his describe rnn architecture like relation model. My question is: description of types RNN by Ian Goodfellow are equal to Andrej Karpathy? If not what it the diffrence beetween this descriptions?

Answer 1

They match up fairly well.

The first Goodfellow description is Karpathy's final "many to many" image. The output at each time step is based on the previous hidden state of the net and the input.

The third Goodfellow description directly corresponds to Karpathy's "many to one" image. This model reads an entire input sequence, and then produces one output.

The only difference is that the second description from Goodfellow's text isn't captured by Karpathy's image. Here's my rendition of what that description states.