Nowadays there's a trend towards using architectures of "deep" RNNs i.e. vertically stacked RNNs. RNN chapter from Bengio's bookThese networks seem to work well in practice.

What's the intuition around using vertically stacked layers of RNNs (beyond the obvious fact that they increase the capacity by increasing the # parameters)?


1 Answer 1


All neural networks can increase expressiveness and representational capacity by stacking layers. Each later layer can learn to non-linearly weigh the earlier layers. These non-linearities allow any function to be approximated. In the case of Recurrent Neural Network (RNN), it is functions over time. Stacked RNNs have increased abilities to learn functions over time.

  • $\begingroup$ How do you reconcile this with the universal approximation theorem saying that only one layer is necessary? $\endgroup$
    – Dave
    May 1, 2020 at 23:56
  • $\begingroup$ If you have only have one layer, you'll need a very wide network in order to learn complex functions. $\endgroup$ May 2, 2020 at 0:31
  • $\begingroup$ I’d say that it worth mentioning...that you get more ability to fit per extra parameter if you stack layers than if you just make one very wide layer. $\endgroup$
    – Dave
    May 2, 2020 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.