# What is the role of $W_{ax}, W_{aa}, W_{ay}$ in forward propagation in RNN? Are they hyperparameters? Why are they needed?

In RNN introduction in Coursera sequence model course, the following formula for forward propagation in RNN was introduced. What exactly is the role of $$W_{ax}, W_{aa}, W_{ay}$$? What do they do?

In the lecture, it was told that:

• $$W_{ax}$$: parameter governing connection from $$x$$ to hidden layer (not sure what exactly does that mean: what happens if it is not governed?)

• $$W_{aa}$$: governing activations (Why govern activations? What happens if it is not governed?)

• $$W_{ay}$$ governs output prediction (What is the point of governing that? What happens if it is not governed?)

In standard neural network these were the formula of forward propogation

$$z_1 = w_1 X_1+b_1 \\ A_1 = g(Z_1)$$ Consider there were only 4 layers then last layer $$z_4 = w_4X_4 + b_4 \\ A_4 \text{ or } \hat{y} = g(z4)$$ I'm able to correlate these equation with $$a^{<1>}$$ in RNN but I am unable to correlate the role of $$W_{ya}$$ present in $$\hat{y}^{<1>}$$ highlighted in yellow. Please explain in very simple terms with an example on what is the job of $$W_{ax}, W_{aa}, W_{ay}$$ in forward pass in standard RNN.

Don't get hung up on the word "govern" here. $$W_{ax}$$, $$W_{ay}$$ and $$W_{aa}$$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $$W_{aa}$$):

• $$W_{ax}$$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
• $$W_{ay}$$ are the weights from your last hidden layer to the output layer (just as they are in feedforward networks)
• $$W_{aa}$$ are the weights applied to the hidden state when it is fed from $$t$$ to $$t+1$$ (this is what you do not have in feedforward networks since they do not propagate through time)

You can also read more about this in the respective chapter of the deep learning book which, I think, provides a good explanation of RNNs.

In a feedforward network you would have $$\hat{y} = g(W_{ay}\text{ }a+b_y)$$ while your RNN has $$\hat{y}^{} = g(W_{ay}\text{ }a^{}+b_y)$$ (with $$i$$ being an index for the time step). There is really nothing new here with regards to the output layer in an RNN. In the below image you see to which connections the weights are applied in a feedforward net and in an RNN (I omitted the bias for simplicity):
• @Aj_MLstater I think you're misreading the formula. $\hat{y} = g(W_{ay}\text{ }a+b_y)$ does not include the variable $y$. It only uses $y$ as an index for the weight $W_{ay}$. Which is just a way to name the weight connecting the final hidden layer to the output layer (you could also change its name to sth else like $W_{out}$ - it's just a name). Because, as you said yourself, $y$ does not play a role in the feedforward process. It just comes into play after when calculating the loss. – Sammy Jan 21 '20 at 22:39