Hot answers tagged

109

GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM- The GRU controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control. GRU is relatively new, and from ...


66

*To complement already great answers above. From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks). GRUs are simpler and thus easier to modify, for example adding new gates in case of additional input to the network. It's just less code in general. LSTMs ...


21

FULL GRU Unit $ \tilde{c}_t = \tanh(W_c [G_r * c_{t-1}, x_t ] + b_c) $ $ G_u = \sigma(W_u [ c_{t-1}, x_t ] + b_u) $ $ G_r = \sigma(W_r [ c_{t-1}, x_t ] + b_r) $ $ c_t = G_u * \tilde{c}_t + (1 - G_u) * c_{t-1} $ $ a_t = c_t $ LSTM Unit $ \tilde{c}_t = \tanh(W_c [ a_{t-1}, x_t ] + b_c) $ $ G_u = \sigma(W_u [ a_{t-1}, x_t ] + b_u) $ $ G_f = \sigma(W_f [...


14

This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better. GRU exposes the complete memory unlike LSTM, so applications which that acts as advantage might be helpful. Also, adding onto why to use GRU - it is computationally easier than LSTM since it has only 2 gates and if it's performance is on par with LSTM,...


3

This flag is used to have truncated back-propagation through time: the gradient is propagated through the hidden states of the LSTM across the time dimension in the batch and then, in the next batch, the last hidden states are used as input states for the LSTM. This allows the LSTM to use longer context at training time while constraining the number of ...


1

Looking more, the coursera article and proposed architecture is excellent, i would start with it as-is. The tensorflow dataset is also valuable (impressed by its public existence). Summary: The GRU network should be fed with sequencial 1D vectors (row-by-row capturing the event evolving in time t,t+1,t+2) but the tensorflow's classical CNN with whole 2D ...


1

The article [coursera] use properly the GRU so, these changes would be neccesary: Architecture: Use only one GRU layer having in front the Conv1D not Conv2D (super trivial). The input should be a fixed length 1D vector, (single row extracted from the 2D spectra at t0, vector of 5511 in the article). Then feed the vector from next row at t+1, then next row ...


1

I think your network's design is fine but need change the way you feed training data. In essence you try convert a classic CNN (having "image" data, but spectrograms) to a sequential time-domain type of network (recurrent one, having GRU or LSTM at the heart). To be able to do that you have to reconsider the way you feed the input data: Input data ...


1

The initial weights of h for GRU and h,c for LSTM are are often set to zeros, setting random weights is also an option. Also people have tried to learn the initial hidden states. Since the hidden states are updated with every cell, if your sequences are long enough, it would not make a big difference how you initialize the hidden states.


1

This is probably because the deltas you are trying to predict are less than 1, so your loss function (I’m assuming MSE) isn’t working as expected. Squaring an error less than 1 will make it even smaller, so your model is not currently motivated to leave the cosy local minimum of the naïve strategy of always predicting delta as zero. I recommend rescaling ...


Only top voted, non community-wiki answers of a minimum length are eligible