Why is a RNN inherently better for Time series than normal NN?

Question

Similar to this question but I would like further clarification.

I understand that in abstract, RNNs can process inputs recursively and feed some state of memory through the recursion to have a sense of context and order.

However, why can a normal NN not achieve this? The input vector is inherently ordered. For example in language modelling, one might define a length for each token, and input a series of these tokens into the standard NN, and the NN could work out by itself that these are ordered and infer context in order to output its best prediction of the next token.

Is the benefit that the input be 1 token at a time and so the RNN needs less complexity? Or is there something about RNN's that a normal NN simply cannot achieve? Or are they just more effective at interpreting the ordered nature of the input? If so why?

I suppose I could generalise the question to - why do we need any kind of specific NN? Can a normal NN not approximate any function? Surely it could therefore learn any behaviour that some specific kind of NN exhibits?

Answer 1

Certainly, the original language model by Bengio et al, 2003 worked with "normal NNs". However, they worked by simply concatenating word embeddings and then applying the transformation $y = b + Wx + U \mathsf{tanh}(d + Hx)$.

This kind of language model presents some problems:

• Scalability: the longer you want the context window to be, the larger the matrix multiplications you need.
• Training efficiency: you cannot train for all the output words of a sequence, that is, you can only train one output word at a time. Therefore, each sequence in the training data leads to multiple training data points (one per each possible location of the context window within the sequence).
• Order-dependent representations (lack of generalization): the representations learned for a word appearing at a specific position within the context window cannot be generalized to other positions.

Modern language models are all RNNs or Transformers, which certainly don't have the aforementioned problems (although Transformers do have problems scaling the context window due to the quadratic memory requirements of attention).