183
$\begingroup$

The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).

Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ A GRU is slightly less complex but is approximately as good as an LSTM performance-wise. An implementation in TensorFlow is found here: data-blogger.com/2017/08/27/gru-implementation-tensorflow. $\endgroup$
    – www.data-blogger.com
    Commented Aug 27, 2017 at 12:28
  • $\begingroup$ GRUs are generally used when you do have long sequence training samples and you want a quick and decent accuracy and maybe in cases where infrastructure is an issue. LSTMs are preferred when sequence lengths are more and some good context is there. LSTMs when trained with more data give you better results than GRUs. $\endgroup$
    – Subir Verma
    Commented Oct 18, 2020 at 8:03
  • 3
    $\begingroup$ @SubirVerma, doesn't "long sequence training samples" and "sequence lengths are more" mean the same thing? $\endgroup$
    – Naveen Reddy Marthala
    Commented Jun 26, 2022 at 11:30

6 Answers 6

Reset to default
120
$\begingroup$

GRUs and LSTMs utilize different approaches toward gating information to prevent the vanishing gradient problem. Here are the main points comparing the two:

  • The GRU unit controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control.
  • GRUs are relatively new, and in my experience, their performance is on par with LSTMs, but computationally more efficient (as pointed out, they have a less complex structure). For that reason, we are seeing it being used more and more.

For a detailed description, you can explore this research paper on Arxiv. The paper explains all this brilliantly.

You can also explore these blogs for a better idea:

Hope that helps!

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ In addition to your answer there is a nice paper evaluating the performance between GRU and LSTM and their various permutations "An empirical exploration of recurrent network architectures" by Google $\endgroup$
    – tastyminerals
    Commented Jun 10, 2017 at 18:11
  • $\begingroup$ @abhishek I found your answer is a bit counter intuitive - how GRU "without having memory unit" can have a better performance - perhaps I should read the other papers $\endgroup$
    – user702846
    Commented Jan 22, 2020 at 13:45
73
$\begingroup$

*To complement already great answers above.

  • From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks).

  • GRUs are simpler and thus easier to modify, for example adding new gates in case of additional input to the network. It's just less code in general.

  • LSTMs should in theory remember longer sequences than GRUs and outperform them in tasks requiring modeling long-distance relations.

*Some additional papers that analyze GRUs and LSTMs.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Good balance of specific precise details while yet concise. $\endgroup$
    – WestCoastProjects
    Commented Feb 20, 2021 at 16:57
24
$\begingroup$

FULL GRU Unit

$ \tilde{c}_t = \tanh(W_c [G_r * c_{t-1}, x_t ] + b_c) $

$ G_u = \sigma(W_u [ c_{t-1}, x_t ] + b_u) $

$ G_r = \sigma(W_r [ c_{t-1}, x_t ] + b_r) $

$ c_t = G_u * \tilde{c}_t + (1 - G_u) * c_{t-1} $

$ a_t = c_t $

LSTM Unit

$ \tilde{c}_t = \tanh(W_c [ a_{t-1}, x_t ] + b_c) $

$ G_u = \sigma(W_u [ a_{t-1}, x_t ] + b_u) $

$ G_f = \sigma(W_f [ a_{t-1}, x_t ] + b_f) $

$ G_o = \sigma(W_o [ a_{t-1}, x_t ] + b_o) $

$ c_t = G_u * \tilde{c}_t + G_f * c_{t-1} $

$ a_t = G_o * tanh(c_t) $

As can be seen from the equations LSTMs have a separate update gate and forget gate. This clearly makes LSTMs more sophisticated but at the same time more complex as well. There is no simple way to decide which to use for your particular use case. You always have to do trial and error to test the performance. However, because GRU is simpler than LSTM, GRUs will take much less time to train and are more efficient.

Credits:Andrew Ng

Improve this answer
$\endgroup$
1
  • $\begingroup$ On the next slide after the hand written ones, the last equation is different: $a^{<t>} = \Gamma_o \odot tanh\left(\tilde c^{<t>}\right)$. This formula is confirmed correct here. $\endgroup$
    – Tom Hale
    Commented Mar 25, 2019 at 8:04
16
$\begingroup$

This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better.

  • GRU exposes the complete memory unlike LSTM, so applications which that acts as advantage might be helpful. Also, adding onto why to use GRU - it is computationally easier than LSTM since it has only 2 gates and if it's performance is on par with LSTM, then why not?
  • This paper demonstrates excellently with graphs the superiority of gated networks over a simple RNN but clearly mentions that it cannot conclude which of the either are better. So, if you are confused as to which to use as your model, I'd suggest you to train both and then get the better of them.
Improve this answer
$\endgroup$
-2
$\begingroup$

GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance.

Improve this answer
$\endgroup$
1
  • 34
    $\begingroup$ please support the performance claim with fair references $\endgroup$
    – Kari
    Commented May 21, 2018 at 3:57
-2
$\begingroup$

Actually, the key difference comes out to be more than that: Long-short term (LSTM) perceptrons are made up using the momentum and gradient descent algorithms. When you reconcile LSTM perceptrons with their recursive counterpart RNNs, you come up with GRU which is really just a generalized recurrent unit or Gradient Recurrent Unit (depending on the context) that more closely integrates the momentum and gradient descent algorithms. Were I you, I'd do more research on AdamOptimizers.

GRU is an outdated concept by the way. However, I can understand you researching it if you want moderate-advanced in-depth knowledge of TF.

Improve this answer
$\endgroup$
2
  • 24
    $\begingroup$ I'm curious. Could you explain why GRU is an outdated concept? $\endgroup$
    – random_user
    Commented Nov 16, 2018 at 18:01
  • 3
    $\begingroup$ If you make a claim that the primary topic of the question is "an outdated concept" you really should back that up with references (as mentioned in prior comment) $\endgroup$
    – WestCoastProjects
    Commented Feb 20, 2021 at 16:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.