Skip to main content
added 764 characters in body
Source Link
tony
  • 591
  • 2
  • 5

The equations are almost the same. First of all, justthey are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referedreferred to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:

$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$

and if we use the same order as Colah's equation:

Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$

Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$

So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation. If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)

The equations are the same, just written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are refered to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

The equations are almost the same. First of all, they are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referred to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:

$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$

and if we use the same order as Colah's equation:

Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$

Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$

So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation. If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)

added 6 characters in body
Source Link
tony
  • 591
  • 2
  • 5

The equations are the same, just written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are refered to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

The equations are the same, just written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are refered to as $W$.

For example you could these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

The equations are the same, just written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are refered to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Source Link
tony
  • 591
  • 2
  • 5

The equations are the same, just written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are refered to as $W$.

For example you could these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$