Reading On Multiplicative Integration with Recurrent Neural Networks

Despite of their varying characteristics, most of them(RNNs) share a common computational building block, described by the following equation: $φ(W_x + U_z + b)$, where $x ∈ R_n$ and $z ∈ R_m$ are state vectors coming from different information sources, $W ∈ R_{d×n}$ and $U ∈ R_{d×m}$ are state-to-state transition matrices, and $b$ is a bias vector.

Don't get what meaning of $W_x$ and $U_z$. I know that $W$ is typicaly for weights... what does this equation mean?

up vote 4 down vote accepted

If you look deeper in LSTMs or GRUs, we observe that the gates(input, output, cell or forget based on the RNN) are calculated using an equation like you specified.

Example, according to deep learning tutorial of lstm, it=sigma(Wi xt + Ui ht-1 + bi) In this, h is the hidden state vector and x is the input state vector as specified and W and U are the corresponding weights for the input gate it. SImilarly, there are gates for output and forget. So in the paper, they recall a gist of RNNs and sum it up as a general equation. It is a common computational block in RNNs despite their minor differences.

Refer Colah's blog or wildml, I think they are one of the best to understand RNNs.

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.