This question gets at some very important qualities of RNNs and DNNs in general. I'll answer each of your sub-questions, though not in the same order (I'll try to highlight where I do)
Parameter Sharing
First, the most important quality of RNNs is called parameter sharing. Sequential data is typically input into separate layers. For a length 20 input, an RNN network would have 20 layers. The same internal parameters are used for each layer, so all 20 layers use the same weights $W$ and bias $b$. Compare this to a multilayer perceptron which would have 20 separate weights and biases.
Parameter sharing has several benefits:
- We now have many fewer parameters. There is 1 repeated block instead of 20 separate layers. A 20x reduction!
- This effectively multiplies the training data. The recurrent layer gets to learn from every word in a single example sentence whereas each layer in an MLP learns from a single word per sentence.
- Our network is now much more flexible. We can train on sentences up to 20 words and then generalize to a 25 word sentence by adding more steps or by using a dynamic RNN
Network architecture
You ask about tanh and sigmoid activations. To answer this we have to talk about specific RNN architectures. The simple RNN discussed above has a single activation. Simple RNNs tend to create the problem of vanishing gradients (or exploding!) due to the repeated application of the same weights and activation function.
Gated RNN blocks (like GRUs and LSTMs) use gating mechanisms to pass activations into and out of memory states and to combine memory states with input to generate the output of each block. Because of this, gates can stop the gradient from propagating backward. Sigmoid is a common activation function for gates because it squashes activations to (0,1)---0 completely stops the activation and 1 lets it pass through. Any decent activation function with a similar squashing profile works, though. Anecdotally, hard sigmoid is quite common these days.
In addition to the gates, gated RNN blocks have an internal state for which the activation varies quite a bit. Because gating limits gradient backprop, we have a lot of flexibility on this activation. It need not be squashing for instance, and this is where the rectifying activations (relu, elu, islu, etc.) are often seen. Tanh is a perfectly sensible choice as well.
Regarding biases and weights, each activation in an RNN cell typically has its own weight and bias. So a GRU has 3 activations (hidden, update, and reset) and each has its own weight and bias. Though, recall that as an RNN, each of these are re-used for every timestep.
Backward Pass
That covers the forward pass pretty well but you also ask an important question about how error propagates backward. There are two methods for approaching this.
Teacher Forcing
For RNNs that output a prediction at each time step (such as predicting the outcome of following steps, translation, or phoneme recognition), teacher forcing is a method for isolating each step of the RNN. By removing these dependencies, Teacher Forcing allows RNN to use conventional backprop with the chain rule.
But how does it work? Teacher forcing networks have separate train and test architectures. For training, at each timestep $t$, the input $x_t$ is concatenated with the previous target, $y_{t-1}$. Imagine this for a network tasked with predicting the following character. The network has just tried to predict the character for the previous timestep. But we instead use the observed character at that timestep (we know this because we're in the training phase). Thus the error at timestep $t$ depends only on the observed value at $t-1$ and the input at $t$. We've thus removed any connections through time from the network.
At test, we don't know the true value at each timestep so we replace $y_{t-1}$ with the output of the previous layer $o_{t-1}$. In this case, temporal connections have returned but only for the test phase.
Back Propagation Through Time
But we don't have to resort to Teacher Forcing. Back propogation through time allows us to apply the backprop algorithm to RNNs. Consider the network with $n$ timesteps and an input $x$, hidden state $h$, output $o$, and observed value $y$ for each timestep.
BPTT works in the following steps.
- Compute the gradient $\nabla_{o_t}$ for each $o_t, y_t$ pair. (This may be done all at once.)
- Compute the gradient $\nabla_{h_t}$ for each timestep, beginning with the last timestep and iteratively working backward. (This must be done one at a time.)
- This gives us $n$ edges for each internal parameter of our RNN. The trick to updating the parameters is finding the gradient contribution for each timestep (eg $\nabla_{W_t}$ even though we only have one $W$) and then summing those gradients to update the internal parameters.
Further reading
I highly recommend chapter 10 of Goodfellow, Bengio, and Courville's Deep Learning for more information on RNNs. Additionally Graves' RNN book is fantstic for higher-level detail
