not quite sure about the difference between RNN and feed forward neural net

Question

I'm a bit confused after reading this paper: https://arxiv.org/abs/1705.09851

on page 22, the author writes

response:

\begin{equation} Y = softmax(Z^{L-1}) \end{equation}

and hidden state

\begin{equation} Z^\ell = max(W^\ell *Z^{\ell-1} + b^\ell, 0) \end{equation}

which is a relu

But, to me, this looks like a regular feed forward neural net- you multiply your input by a matrix, add a bias unit, then activate. Alternatively, your hidden layer is equal to the activation of the sum of a bias and the previous hidden layer times a weight matrix.

What am I missing?

Answer 1

The authors state that's the formulation for a feed forward deep learner, so you're exactly right. The two equations at the bottom of the page are where they formulate their recurrent neural net

The response is

$\hat{Y} = \text{softmax}(W^2Z_t+b^2) $

and the hidden state is

$Z_{t-j} = \text{tanh}(W^1[Z_{t-j-1},X_{t-j}] + b^1), j \in\{k,...,0\}$

The authors punt to this paper for implementation details, but the recurrent nature here comes from directly articulating your hidden state off of your most recent hidden state $Z_{t-j-1}$ (which involves the input $X_{t-j-1}$ from the last time step).

Hope this helps!