Advantages of Recurrent Neural Networks over basic Artificial Neural Networks

Question

I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me:

The unfodling process thus introduces two major advantages:

1. Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition from one state to another state, rather then specified in terms of variable-length history of states.
2. It is possible to use same transition function f with same parameters at every time step.

These two factors make it possible to learn a single model f that operates on all time steps and all sequence lengths, rather then needing to learn a separate model for all possible time steps. Learning a single shared model allows generalization to sequence lengths that did not appear in the training set, and enables the model to be estimated with far fewer training examples than would be required without parameter sharing.

I understand the 2nd advantage. Because the computations are recurrent, the input besides the current element in the sequence is the output of the previous hidden state which has the same structure as the current hidden state, thus the shared parameters.

But I do not understand the first advantage. I cannot visualize it or mathematically prove it, or at least I dont know how. Can anyone help me with this one? And if anyone has anything else to add on the difference ( advantages ) of RNN over ANN I would really appreciate it. Thanks in advance!

Answer 1

okey, I will try to explain them as easy as possible.

Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition from one state to another state, rather then specified in terms of variable-length history of states.

I will use a simple example for simplification but it does not lead to losing generalisation. Suppose that your task is to add to binary numbers. Numbers are stored bitwise in memory and usually for applying any arithmetic operations, they should have a same size. What you need for adding two numbers is to learn how to add zeros and ones and when to output carray to the next step if you use sequential learning approaches, like RNNs. In this case, it is not really important that the two numbers do not have the same shape due to the fact that they both can be resized to the biggest size by adding zeros to their most significant bits. After resizing the inputs to have the same shape you have two options, to use MLPs or RNNs. If you use MLPs what the network learns is entirely different from what an RNN learns. The former does not learn the transition of carray at least as the way RNN learns. Another difference is that your MLP always will be restricted to the size which it was trained while the RNN model will be able to add two numbers with even more bits. To explain it why, for MLPs, all inputs which are connected to the hidden layers or maybe output layers, have weight. Consequently, increasing the number of inputs will lead to more weights which are not trained yet. You are not allowed to input a signal which its size is not equal to the input size of the MLPs. On the contrary, RNNs are exploited in a different way.

First, you should know RNNs better. Try to think of hidden layers of anRNN. They are like usual MLPs. Their difference is that for each node, the inputs come from the previous step's outputs and the current time's inputs. Bear in mind that the outcomes of the previous time step are not coming from previous neurons. The reason and I guess the main answer to your question is that each RNN is repeated for each input in time $t$. It means you have just an MLP which is used for all time step. Suppose you are at the middle of the calculations. Two inputs are $1$ and $1$. You RNN takes them and the carray value and outputs $1$ as the result of time step $t$ and outputs $1$ to the next step as carray. For the next time step the same RNN is again used and takes the inputs alongside the carry which is coming from the previous step's outputs and outputs the corresponding outputs and carries.

Due to the nature of RNNs which just take the inputs of time step $t$, they are capable of dealing with signals with different lengths. The reason is that The RNN is used for each time step. This behaviour is usually called unfolding the network.

Answer 2

The main advantage of RNN over ANN is that RNN can model sequence of data (i.e. time series) so that each sample can be assumed to be dependent on previous ones. On the contrary, ANN can not model sequence of data. So, ANN is useful if only each sample is assumed to be independent of previous and next ones (akn as iid assumption).