I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs.

I've been looking at various posts now related to recurrent neural network, and I understand the concept behind it, but I fail to understand certain parts of the logic behind the RNN.

Here are four main questions I have:

  1. How does back-propagation work in recurrent neural network?
  2. Are the weights that lead from the input to the hidden nodes the same for every other step? What about the weights from the hidden nodes to the outputs?
  3. How do biases exactly work in a recurrent neural network?
  4. Why is tanh function usually used instead of sigmoid function as the activtion function?

I realize some of these questions are very basic, but I guess the basics are exactly what I need right now.

Even links to related videos or posts will be extremely helpful, and so will google keywords that show the right results. These four questions are hindering me from understanding sample Python codes, so I really need some help.

  • $\begingroup$ "the same number of outputs as inputs". Wait, what? Be careful with this - it's not often that you come across a NN where your statement would be true. These numbers can vary wildly and I can't think of the last time I worked on a NN where my inputs and outputs matched exactly. Your inputs are whatever you want them to be based on available features, your outputs depends on the answer you are trying to get (binary, classification, etc). Very rare to have those two match up. $\endgroup$ Feb 5, 2018 at 16:04

1 Answer 1


This question gets at some very important qualities of RNNs and DNNs in general. I'll answer each of your sub-questions, though not in the same order (I'll try to highlight where I do)

Parameter Sharing

First, the most important quality of RNNs is called parameter sharing. Sequential data is typically input into separate layers. For a length 20 input, an RNN network would have 20 layers. The same internal parameters are used for each layer, so all 20 layers use the same weights $W$ and bias $b$. Compare this to a multilayer perceptron which would have 20 separate weights and biases.

Parameter sharing has several benefits:

  • We now have many fewer parameters. There is 1 repeated block instead of 20 separate layers. A 20x reduction!
  • This effectively multiplies the training data. The recurrent layer gets to learn from every word in a single example sentence whereas each layer in an MLP learns from a single word per sentence.
  • Our network is now much more flexible. We can train on sentences up to 20 words and then generalize to a 25 word sentence by adding more steps or by using a dynamic RNN

Network architecture

You ask about tanh and sigmoid activations. To answer this we have to talk about specific RNN architectures. The simple RNN discussed above has a single activation. Simple RNNs tend to create the problem of vanishing gradients (or exploding!) due to the repeated application of the same weights and activation function.

Gated RNN blocks (like GRUs and LSTMs) use gating mechanisms to pass activations into and out of memory states and to combine memory states with input to generate the output of each block. Because of this, gates can stop the gradient from propagating backward. Sigmoid is a common activation function for gates because it squashes activations to (0,1)---0 completely stops the activation and 1 lets it pass through. Any decent activation function with a similar squashing profile works, though. Anecdotally, hard sigmoid is quite common these days.

In addition to the gates, gated RNN blocks have an internal state for which the activation varies quite a bit. Because gating limits gradient backprop, we have a lot of flexibility on this activation. It need not be squashing for instance, and this is where the rectifying activations (relu, elu, islu, etc.) are often seen. Tanh is a perfectly sensible choice as well.

Regarding biases and weights, each activation in an RNN cell typically has its own weight and bias. So a GRU has 3 activations (hidden, update, and reset) and each has its own weight and bias. Though, recall that as an RNN, each of these are re-used for every timestep.

Backward Pass

That covers the forward pass pretty well but you also ask an important question about how error propagates backward. There are two methods for approaching this.

Teacher Forcing

For RNNs that output a prediction at each time step (such as predicting the outcome of following steps, translation, or phoneme recognition), teacher forcing is a method for isolating each step of the RNN. By removing these dependencies, Teacher Forcing allows RNN to use conventional backprop with the chain rule.

But how does it work? Teacher forcing networks have separate train and test architectures. For training, at each timestep $t$, the input $x_t$ is concatenated with the previous target, $y_{t-1}$. Imagine this for a network tasked with predicting the following character. The network has just tried to predict the character for the previous timestep. But we instead use the observed character at that timestep (we know this because we're in the training phase). Thus the error at timestep $t$ depends only on the observed value at $t-1$ and the input at $t$. We've thus removed any connections through time from the network.

At test, we don't know the true value at each timestep so we replace $y_{t-1}$ with the output of the previous layer $o_{t-1}$. In this case, temporal connections have returned but only for the test phase.

Back Propagation Through Time

But we don't have to resort to Teacher Forcing. Back propogation through time allows us to apply the backprop algorithm to RNNs. Consider the network with $n$ timesteps and an input $x$, hidden state $h$, output $o$, and observed value $y$ for each timestep.

BPTT works in the following steps.

  1. Compute the gradient $\nabla_{o_t}$ for each $o_t, y_t$ pair. (This may be done all at once.)
  2. Compute the gradient $\nabla_{h_t}$ for each timestep, beginning with the last timestep and iteratively working backward. (This must be done one at a time.)
  3. This gives us $n$ edges for each internal parameter of our RNN. The trick to updating the parameters is finding the gradient contribution for each timestep (eg $\nabla_{W_t}$ even though we only have one $W$) and then summing those gradients to update the internal parameters.

Further reading

I highly recommend chapter 10 of Goodfellow, Bengio, and Courville's Deep Learning for more information on RNNs. Additionally Graves' RNN book is fantstic for higher-level detail


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.