# not quite sure about the difference between RNN and feed forward neural net

I'm a bit confused after reading this paper: https://arxiv.org/abs/1705.09851

on page 22, the author writes

response:

$$\begin{equation} Y = softmax(Z^{L-1}) \end{equation}$$

and hidden state

$$\begin{equation} Z^\ell = max(W^\ell *Z^{\ell-1} + b^\ell, 0) \end{equation}$$

which is a relu

But, to me, this looks like a regular feed forward neural net- you multiply your input by a matrix, add a bias unit, then activate. Alternatively, your hidden layer is equal to the activation of the sum of a bias and the previous hidden layer times a weight matrix.

What am I missing?

The authors state that's the formulation for a feed forward deep learner, so you're exactly right. The two equations at the bottom of the page are where they formulate their recurrent neural net

The response is

$$\hat{Y} = \text{softmax}(W^2Z_t+b^2)$$

and the hidden state is

$$Z_{t-j} = \text{tanh}(W^1[Z_{t-j-1},X_{t-j}] + b^1), j \in\{k,...,0\}$$

The authors punt to this paper for implementation details, but the recurrent nature here comes from directly articulating your hidden state off of your most recent hidden state $$Z_{t-j-1}$$ (which involves the input $$X_{t-j-1}$$ from the last time step).

Hope this helps!

• thanks. arxiv version of the paper you mention: arxiv.org/abs/1707.05642 Nov 5, 2018 at 21:34
• for those that want more reading- it looks like Dixon follows this pretty closely, but changes some notations: karpathy.github.io/2015/05/21/rnn-effectiveness Nov 5, 2018 at 21:35
• Another important aspect of recurrent nets vs feed forward nets is that they share weights. Nov 5, 2018 at 21:38