Hot answers tagged

89

GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM- The GRU controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control. GRU is relatively new, and from ...


53

*To complement already great answers above. From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks). GRUs are simpler and thus easier to modify, for example adding new gates in case of additional input to the network. It's just less code in general. LSTMs ...


15

FULL GRU Unit $ \tilde{c}_t = \tanh(W_c [G_r * c_{t-1}, x_t ] + b_c) $ $ G_u = \sigma(W_u [ c_{t-1}, x_t ] + b_u) $ $ G_r = \sigma(W_r [ c_{t-1}, x_t ] + b_r) $ $ c_t = G_u * \tilde{c}_t + (1 - G_u) * c_{t-1} $ $ a_t = c_t $ LSTM Unit $ \tilde{c}_t = \tanh(W_c [ a_{t-1}, x_t ] + b_c) $ $ G_u = \sigma(W_u [ a_{t-1}, x_t ] + b_u) $ $ G_f = \sigma(W_f [...


12

This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better. GRU exposes the complete memory unlike LSTM, so applications which that acts as advantage might be helpful. Also, adding onto why to use GRU - it is computationally easier than LSTM since it has only 2 gates and if it's performance is on par with LSTM,...


2

This flag is used to have truncated back-propagation through time: the gradient is propagated through the hidden states of the LSTM across the time dimension in the batch and then, in the next batch, the last hidden states are used as input states for the LSTM. This allows the LSTM to use longer context at training time while constraining the number of ...


1

The initial weights of h for GRU and h,c for LSTM are are often set to zeros, setting random weights is also an option. Also people have tried to learn the initial hidden states. Since the hidden states are updated with every cell, if your sequences are long enough, it would not make a big difference how you initialize the hidden states.


1

Actually, the key difference comes out to be more than that: Long-short term (LSTM) perceptrons are made up using the momentum and gradient descent algorithms. When you reconcile LSTM perceptrons with their recursive counterpart RNNs, you come up with GRU which is really just a generalized recurrent unit or Gradient Recurrent Unit (depending on the context) ...


1

GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance.


Only top voted, non community-wiki answers of a minimum length are eligible