I'm a bit confused after reading this paper: https://arxiv.org/abs/1705.09851

on page 22, the author writes


\begin{equation} Y = softmax(Z^{L-1}) \end{equation}

and hidden state

\begin{equation} Z^\ell = max(W^\ell *Z^{\ell-1} + b^\ell, 0) \end{equation}

which is a relu

But, to me, this looks like a regular feed forward neural net- you multiply your input by a matrix, add a bias unit, then activate. Alternatively, your hidden layer is equal to the activation of the sum of a bias and the previous hidden layer times a weight matrix.

What am I missing?


1 Answer 1


The authors state that's the formulation for a feed forward deep learner, so you're exactly right. The two equations at the bottom of the page are where they formulate their recurrent neural net

The response is

$\hat{Y} = \text{softmax}(W^2Z_t+b^2) $

and the hidden state is

$Z_{t-j} = \text{tanh}(W^1[Z_{t-j-1},X_{t-j}] + b^1), j \in\{k,...,0\}$

The authors punt to this paper for implementation details, but the recurrent nature here comes from directly articulating your hidden state off of your most recent hidden state $Z_{t-j-1}$ (which involves the input $X_{t-j-1}$ from the last time step).

Hope this helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.