The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).

Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM?

GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM-

  • The GRU unit controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control.
  • GRU is relatively new, and from my perspective, the performance is on par with LSTM, but computationally more efficient (less complex structure as pointed out). So we are seeing it being used more and more.

For detail description you can explore this Research Paper - Arxiv.org. The paper explains all this brilliantly.

Plus, you can also explore these blogs for a better idea-

Hope it helps!

*To complement already great answers above.

  • From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks).

  • GRUs are simpler and thus easier to modify, for example adding new gates in case of additional input to the network. It's just less code in general.

  • LSTMs should in theory remember longer sequences than GRUs and outperform them in tasks requiring modeling long-distance relations.

*Some additional papers that analyze GRUs and LSTMs.

This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better.

  • GRU exposes the complete memory unlike LSTM, so applications which that acts as advantage might be helpful. Also, adding onto why to use GRU - it is computationally easier than LSTM since it has only 2 gates and if it's performance is on par with LSTM, then why not?
  • This paper demonstrates excellently with graphs the superiority of gated networks over a simple RNN but clearly mentions that it cannot conclude which of the either are better. So, if you are confused as to which to use as your model, I'd suggest you to train both and then get the better of them.

GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance.

  • 4
    please support the performance claim with fair references – Kari May 21 at 3:57

FULL GRU Unit

𝑐 ̃<𝑡>=tanh⁡(𝑊𝑐 [G𝑟∗𝑐<𝑡−1>, 𝑥<𝑡> ]+𝑏𝑐)

G𝑢=𝜎⁡(𝑊𝑢 [ 𝑐<𝑡−1>, 𝑥(<𝑡>) ]+𝑏𝑢)

G𝑟=𝜎⁡(𝑊𝑟 [ 𝑐<𝑡−1), 𝑥<𝑡> ]+𝑏𝑟)

𝑐<𝑡>= G𝑢⁡∗𝑐 ̃<𝑡> + (1−G𝑢 )∗𝑐<𝑡−1>

𝑎<𝑡>=𝑐<𝑡>

LSTM Unit

𝑐 ̃<𝑡>=tanh⁡(𝑊𝑐 [𝑎(<𝑡−1>),𝑥(<𝑡>) ]+𝑏𝑐)

G𝑢 = 𝜎⁡(𝑊𝑢 [ 𝑎<𝑡−1>, 𝑥<𝑡> ] + 𝑏𝑢)

G𝑓 = 𝜎⁡(𝑊𝑓 [ 𝑎<𝑡−1>, 𝑥<𝑡> ] + 𝑏𝑓)

G𝑜 = 𝜎⁡(𝑊𝑜 [ 𝑎<𝑡−1>, 𝑥<𝑡> ] +𝑏𝑜)

𝑐<𝑡> = ⁡G𝑢∗𝑐 ̃<𝑡> + G𝑓∗𝑐<𝑡−1>

𝑎<𝑡> = G𝑜∗𝑐<𝑡>

As can be seen from the equations LSTMs have a separate update gate and forget gate. This clearly makes LSTMs more sophisticated but at the same time more complex as well. There is no simple way to decide which to use for your particular use case. You always have to do trial and error to test the performance. However, because GRU is simpler than LSTM, GRUs will take much less time to train and are more efficient.

Credits:Andrew Ng

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.