4
$\begingroup$

Nowadays there's a trend towards using architectures of "deep" RNNs i.e. vertically stacked RNNs. RNN chapter from Bengio's bookThese networks seem to work well in practice.

What's the intuition around using vertically stacked layers of RNNs (beyond the obvious fact that they increase the capacity by increasing the # parameters)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

All neural networks can increase expressiveness and representational capacity by stacking layers. Each later layer can learn to non-linearly weigh the earlier layers. These non-linearities allow any function to be approximated. In the case of Recurrent Neural Network (RNN), it is functions over time. Stacked RNNs have increased abilities to learn functions over time.

Improve this answer
$\endgroup$
3
  • $\begingroup$ How do you reconcile this with the universal approximation theorem saying that only one layer is necessary? $\endgroup$
    – Dave
    Commented May 1, 2020 at 23:56
  • $\begingroup$ If you have only have one layer, you'll need a very wide network in order to learn complex functions. $\endgroup$
    – Brian Spiering
    Commented May 2, 2020 at 0:31
  • $\begingroup$ I’d say that it worth mentioning...that you get more ability to fit per extra parameter if you stack layers than if you just make one very wide layer. $\endgroup$
    – Dave
    Commented May 2, 2020 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.