5
$\begingroup$

I think I understand both types of units in terms of just the math.

What I don't understand is, how is it possible in practice for a GRU to perform as well as or better than an LSTM (which is what seems to be the norm)? I don't intuitively get how the GRU is able to make up for the missing cell content. The gates seem to be pretty much the same as an LSTM's gates, but with a missing part. Does it just mean that the cell in an LSTM is actually nearly useless?

Edit: Other questions have asked about the differences between GRU and LSTM. None of them (in my opinion) explain well enough why a GRU works as well as an LSTM even without the memory unit, only that the lack of a memory unit is one of the differences that makes a GRU faster.

$\endgroup$
2
  • $\begingroup$ I saw that thread but I don't think any of the answers fully explain why a GRU is as good, only what the difference is between the two. (for example many answers just say something along the lines of "GRU is faster and doesn't use memory units and performs as well") $\endgroup$
    – rococo
    Mar 15, 2018 at 5:48
  • 1
    $\begingroup$ In simple words, the GRU unit does not have to use a memory unit to control the flow of information like the LSTM unit. It can directly makes use of the all hidden states without any control. GRUs have fewer parameters and thus may train a bit faster or need less data to generalize. But, with large data, the LSTMs with higher expressiveness may lead to better results. In many tasks both architectures yield comparable performance and tuning hyperparameters like layer size is probably more important than picking the ideal architecture. $\endgroup$
    – Aditya
    Mar 15, 2018 at 5:53

2 Answers 2

3
$\begingroup$

GRU and LSTM are two popular RNN variants out of many possible similar architectures motivated by similar theoretical ideas of having a "pass through" channel where gradients do not degrade as much, and a system of sigmoid-based control gates to manage signals passing between time steps.

Even with LSTM, there are variations which may or may not get used, such as adding "peephole" connections between previous cell state and the gates.

LSTM and GRU are the two architectures explored so far that do well across a wide range of problems, as verified by experiment. I suspect, but cannot show conclusively, that there is no strong theory that explains this rough equivalence. Instead we are left with more intuition-based theories or conjectures:

  • GRU has less parameters per "cell", allowing it in theory to generalise better from less examples, at the cost of less flexibility.

  • LSTM has a more sophisticated memory in the form of separating internal cell state from cell output, allowing it to output features useful for a task without needing to memorise those features. This comes at the cost of needing to learn extra gates which help map between state and features.

When considering performance of these architectures in general, you have to allow that some problems will make use of these strengths better, or it may be a wash. For instance, in a problem where forwarding the layer output between time steps is already a good state representation and feature representation, then there is little need for the additional internal state of the LSTM.

In effect the choice between LSTM and GRU is yet another hyperparameter to consider when searching for a good solution, and like most other hyperparameters, there is no strong theory to guide an a priori selection.

$\endgroup$
3
$\begingroup$

As other people pointed, there is no clear superior approach. As commonly happens in ML, more complex approaches are somehow more powerful than simpler ones (they can model more complex relationships) but this always comes at a cost.

In the specific example of GRUs and LSTMs is exactly the same, LSTM have a more complex structure so it has the ability to model more temporal related features. GRUs are simpler (see picture below), so they will perhaps outperform LSTMs when data is scarce or when there is a high risk of overfitting.

Have a look at this paper with a nice empirical evaluation of this two approaches. If you go to the conclusion you will read:

"...we could not make concrete conclusion on which of the two gating units was better."

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.