Similar to this question but I would like further clarification.

I understand that in abstract, RNNs can process inputs recursively and feed some state of memory through the recursion to have a sense of context and order.

However, why can a normal NN not achieve this? The input vector is inherently ordered. For example in language modelling, one might define a length for each token, and input a series of these tokens into the standard NN, and the NN could work out by itself that these are ordered and infer context in order to output its best prediction of the next token.

Is the benefit that the input be 1 token at a time and so the RNN needs less complexity? Or is there something about RNN's that a normal NN simply cannot achieve? Or are they just more effective at interpreting the ordered nature of the input? If so why?

I suppose I could generalise the question to - why do we need any kind of specific NN? Can a normal NN not approximate any function? Surely it could therefore learn any behaviour that some specific kind of NN exhibits?


1 Answer 1


Certainly, the original language model by Bengio et al, 2003 worked with "normal NNs". However, they worked by simply concatenating word embeddings and then applying the transformation $y = b + Wx + U \mathsf{tanh}(d + Hx)$.

This kind of language model presents some problems:

  • Scalability: the longer you want the context window to be, the larger the matrix multiplications you need.
  • Training efficiency: you cannot train for all the output words of a sequence, that is, you can only train one output word at a time. Therefore, each sequence in the training data leads to multiple training data points (one per each possible location of the context window within the sequence).
  • Order-dependent representations (lack of generalization): the representations learned for a word appearing at a specific position within the context window cannot be generalized to other positions.

Modern language models are all RNNs or Transformers, which certainly don't have the aforementioned problems (although Transformers do have problems scaling the context window due to the quadratic memory requirements of attention).

  • $\begingroup$ Okay this makes sense thank you very much. So it’s not that ‘normal’ NNs aren’t capable of something that an RNN is - it’s just RNNs are far more efficient with training, generalisability and scalability. But given a simple multiplayer perceptron with infinite space, time and data - this would be just as effective as an RNN with the same space, time and data? $\endgroup$
    – Pierre
    Commented Apr 26, 2023 at 18:37
  • $\begingroup$ Yes, and also very large computational power and memory. $\endgroup$
    – noe
    Commented Apr 26, 2023 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.