What is the role of $W_{ax}, W_{aa}, W_{ay}$ in forward propagation in RNN? Are they hyperparameters? Why are they needed?

Question

In RNN introduction in Coursera sequence model course, the following formula for forward propagation in RNN was introduced. What exactly is the role of $W_{ax}, W_{aa}, W_{ay}$? What do they do?

In the lecture, it was told that:

• $W_{ax}$: parameter governing connection from $x$ to hidden layer (not sure what exactly does that mean: what happens if it is not governed?)

• $W_{aa}$: governing activations (Why govern activations? What happens if it is not governed?)

• $W_{ay}$ governs output prediction (What is the point of governing that? What happens if it is not governed?)

In standard neural network these were the formula of forward propogation

$$ z_1 = w_1 X_1+b_1 \\ A_1 = g(Z_1) $$ Consider there were only 4 layers then last layer $$ z_4 = w_4X_4 + b_4 \\ A_4 \text{ or } \hat{y} = g(z4) $$ I'm able to correlate these equation with $a^{<1>}$ in RNN but I am unable to correlate the role of $W_{ya}$ present in $\hat{y}^{<1>}$ highlighted in yellow. Please explain in very simple terms with an example on what is the job of $W_{ax}, W_{aa}, W_{ay}$ in forward pass in standard RNN.

Answer 1

Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$):

• $W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
• $W_{ay}$ are the weights from your last hidden layer to the output layer (just as they are in feedforward networks)
• $W_{aa}$ are the weights applied to the hidden state when it is fed from $t$ to $t+1$ (this is what you do not have in feedforward networks since they do not propagate through time)

You can also read more about this in the respective chapter of the deep learning book which, I think, provides a good explanation of RNNs.

EDIT (based on your comment):

In a feedforward network you would have $\hat{y} = g(W_{ay}\text{ }a+b_y)$ while your RNN has $\hat{y}^{<i>} = g(W_{ay}\text{ }a^{<i>}+b_y)$ (with $i$ being an index for the time step). There is really nothing new here with regards to the output layer in an RNN. In the below image you see to which connections the weights are applied in a feedforward net and in an RNN (I omitted the bias for simplicity):

Answer 2

@Sammy below are the 2 images which i have mentioned in comment where im unable to find yhat =Way+b since our goal is to find y ,how can Way(weights * y) come into picture for prediction of Y ?