Explaining GRU equations from two sources

Question

I really confused about how GRU computation really works. I am not really good at math btw.

I read the original source of GRU equations from Cho, et al. (2014), and also a Colah's blog post on the same topic.

And somehow, GRU equations at Colah's blog post Understanding LSTM Networks are different than equations in the Cho's original paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.

These are the equations from Colah's blog:

$$z_t = \sigma(W_z \cdot [h_{t-1},x_t])$$

$$r_t = \sigma(W_r \cdot [h_{t-1},x_t])$$

$$h_t = (1-z_t) \ast h_{t-1} + z_t \ast \tilde{h}_t$$

$$\tilde h_t = tanh(W \cdot [r_t \ast h_{t-1}, x_t]_j)$$

These are the equations from the original paper:

$$z_j = \sigma([\mathbf W_z \mathbf x]_j + [\mathbf U_z \mathbf h_{(t-1)}]_j)$$

$$r_j = \sigma([\mathbf W_r \mathbf x]_j + [\mathbf U_r \mathbf h_{(t-1)}]_j)$$

$$h_j^{(t)} = z_jh_j^{(t-1)} + (1-z_j) \tilde h_j^{(t)}$$

$$\tilde h_j^{(t)} = \phi([\mathbf W \mathbf x]_j + [\mathbf U(\mathbf r \odot h_{(t-1)})]_j)$$

Can someone explain if Colah's equations are the same as the equations in the paper version?

Answer 1

The equations are almost the same. First of all, they are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referred to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:

$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$

and if we use the same order as Colah's equation:

Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$

Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$

So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation. If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)