In RNN introduction in Coursera sequence model course, the following formula for forward propagation in RNN was introduced. What exactly is the role of $W_{ax}, W_{aa}, W_{ay}$? What do they do?

In the lecture, it was told that:

  • $W_{ax}$: parameter governing connection from $x$ to hidden layer (not sure what exactly does that mean: what happens if it is not governed?)

  • $W_{aa}$: governing activations (Why govern activations? What happens if it is not governed?)

  • $W_{ay}$ governs output prediction (What is the point of governing that? What happens if it is not governed?)

enter image description here

In standard neural network these were the formula of forward propogation

$$ z_1 = w_1 X_1+b_1 \\ A_1 = g(Z_1) $$ Consider there were only 4 layers then last layer $$ z_4 = w_4X_4 + b_4 \\ A_4 \text{ or } \hat{y} = g(z4) $$ I'm able to correlate these equation with $a^{<1>}$ in RNN but I am unable to correlate the role of $W_{ya}$ present in $\hat{y}^{<1>}$ highlighted in yellow. Please explain in very simple terms with an example on what is the job of $W_{ax}, W_{aa}, W_{ay}$ in forward pass in standard RNN.


Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$):

  • $W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
  • $W_{ay}$ are the weights from your last hidden layer to the output layer (just as they are in feedforward networks)
  • $W_{aa}$ are the weights applied to the hidden state when it is fed from $t$ to $t+1$ (this is what you do not have in feedforward networks since they do not propagate through time)

You can also read more about this in the respective chapter of the deep learning book which, I think, provides a good explanation of RNNs.

EDIT (based on your comment):

In a feedforward network you would have $\hat{y} = g(W_{ay}\text{ }a+b_y)$ while your RNN has $\hat{y}^{<i>} = g(W_{ay}\text{ }a^{<i>}+b_y)$ (with $i$ being an index for the time step). There is really nothing new here with regards to the output layer in an RNN. In the below image you see to which connections the weights are applied in a feedforward net and in an RNN (I omitted the bias for simplicity):

enter image description here

  • $\begingroup$ thanks ," Way are the weights from your last hidden layer to the output layer (just as they are in feedforward networks) " --> i have never seen this weight Way in feed forward network also can you guide were excatly this weight Way appear in the activation formulas above for feed forward network . $\endgroup$ – Aj_MLstater Jan 20 '20 at 1:29
  • $\begingroup$ @Aj_MLstater I have added a further explanation - hope it helps $\endgroup$ – Sammy Jan 20 '20 at 9:16
  • $\begingroup$ @ Sammy i have a query on what you said about Way in feed forward network -> yhat =g(Way +b) how "y" comes here in "Way" ? , our goal is predict "y" but its been included in the activation how can that happend im confused ? i have only seen yhat=g(z) where z represent wx+b . Can you please tell me how Way is associated in the standard feedforward network ? since yhat was calcuated based on g(z) and not g(Way +b ) . Please see below 2 screenshots of the course where i learnt this and also i searched in web to get this formula yhat =g(Way +b) that you wrote but did not find any . $\endgroup$ – Aj_MLstater Jan 21 '20 at 19:10
  • $\begingroup$ @Aj_MLstater I think you're misreading the formula. $\hat{y} = g(W_{ay}\text{ }a+b_y)$ does not include the variable $y$. It only uses $y$ as an index for the weight $W_{ay}$. Which is just a way to name the weight connecting the final hidden layer to the output layer (you could also change its name to sth else like $W_{out}$ - it's just a name). Because, as you said yourself, $y$ does not play a role in the feedforward process. It just comes into play after when calculating the loss. $\endgroup$ – Sammy Jan 21 '20 at 22:39

where is Way here ?

enter image description here

@Sammy below are the 2 images which i have mentioned in comment where im unable to find yhat =Way+b since our goal is to find y ,how can Way(weights * y) come into picture for prediction of Y ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.