Skip to main content
120 votes

When to use GRU over LSTM?

GRUs and LSTMs utilize different approaches toward gating information to prevent the vanishing gradient problem. Here are the main points comparing the two: The GRU unit controls the flow of ...
Abhishek's user avatar
  • 1,989
73 votes

When to use GRU over LSTM?

*To complement already great answers above. From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks). ...
tastyminerals's user avatar
49 votes
Accepted

Time Series prediction using LSTMs: Importance of making time series stationary

In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training ...
tom's user avatar
  • 2,248
47 votes
Accepted

How to feed LSTM with different input array sizes?

The easiest way is to use Padding and Masking. There are three general ways to handle variable-length sequences: Padding and masking (which can be used for (3)), Batch size = 1, and Batch size > 1,...
Esmailian's user avatar
  • 9,342
41 votes

Validation loss is not decreasing

The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing. Dealing with such a Model: Data Preprocessing: Standardizing and Normalizing the ...
user5722540's user avatar
24 votes

When to use GRU over LSTM?

FULL GRU Unit $ \tilde{c}_t = \tanh(W_c [G_r * c_{t-1}, x_t ] + b_c) $ $ G_u = \sigma(W_u [ c_{t-1}, x_t ] + b_u) $ $ G_r = \sigma(W_r [ c_{t-1}, x_t ] + b_r) $ $ c_t = G_u * \tilde{c}_t + (1 - ...
balboa's user avatar
  • 351
20 votes

What's the difference between the cell and hidden state in LSTM?

In short: Cell state: Long term memory of the model, only part of LSTM models Hidden state: Working memory, part of LSTM and RNN models Additional Information RNN and vanishing/exploding gradients ...
hH1sG0n3's user avatar
  • 2,058
18 votes

Dropout on which layers of LSTM?

I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important ...
Green Falcon's user avatar
  • 14.1k
16 votes

When to use GRU over LSTM?

This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better. GRU exposes the complete memory unlike LSTM, so applications which that acts as advantage ...
Hima Varsha's user avatar
  • 2,346
16 votes
Accepted

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

The input shape for an LSTM must be (num_samples, num_time_steps, num_features). In your example case, combining both cities as input, ...
n1k31t4's user avatar
  • 14.9k
15 votes
Accepted

What is LSTM, BiLSTM and when to use them?

RNN architectures like LSTM and BiLSTM are used in occasions where the learning problem is ...
Green Falcon's user avatar
  • 14.1k
14 votes
Accepted

Sliding window leads to overfitting in LSTM?

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for ...
StatsSorceress's user avatar
14 votes
Accepted

Activation function between LSTM layers

Simple explanation with images We know that an activation is required between matrix multiplications to afford a neural network the ability to model non-linear processes. A classical LSTM cell ...
n1k31t4's user avatar
  • 14.9k
13 votes

So what's the catch with LSTM?

You are right that LSTMs work very well for some problems, but some of the drawbacks are: LSTMs take longer to train LSTMs require more memory to train LSTMs are easy to overfit Dropout is much ...
Imran's user avatar
  • 2,381
13 votes

What is the best method for classification of time series data? Should I use LSTM or a different method?

I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies ...
pcko1's user avatar
  • 3,950
12 votes
Accepted

Prediction interval around LSTM time series forecast

Directly, this is not possible. However, if you model it in a different way you can get out confidence intervals. You could instead of a normal regression approach it as estimating a continuous ...
Jan van der Vegt's user avatar
12 votes
Accepted

Keras LSTM with 1D time series

LSTM layers require data of a different shape. From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should ...
n1k31t4's user avatar
  • 14.9k
11 votes

When to use Stateful LSTM?

As for stateful LSTM and its understanding, refer to here. Quoting an answer from there: "I’m given a big sequence (e.g. Time Series) and I split it into smaller sequences to construct my input ...
pcko1's user avatar
  • 3,950
11 votes

What does the output of model.predict function from Keras mean?

The output of a neural network will never, by default, be binary - i.e. zeros or ones. The network works with continuous values (not discrete) in order to optimise the loss more freely in the ...
n1k31t4's user avatar
  • 14.9k
11 votes
Accepted

What is the job of "RepeatVector" and "TimeDistributed"?

tf.keras.layers.RepeatVector According to the docs : Repeats the input n times. They have also provided an example : ...
Shubham Panchal's user avatar
10 votes
Accepted

Checking for stationarity in LSTM

In principle no you do not need to check for stationarity nor correct for it when you are using an LSTM. The thing about stationarity is that it makes prediction tasks much more efficient, and stable....
Ryan's user avatar
  • 722
10 votes

Binary classification of every time series step based on past and future values

You are facing a very common problem: handling imbalanced data. For neural networks, typical procedures are: Having the proper metrics: global accuracy should not be used. Oversampling the minority ...
ignatius's user avatar
  • 1,668
10 votes
Accepted

Batch Size of Stateful LSTM in keras

I solved the problem this way: I realized that I needed to find the HCF (highest common factor) of both the length of x_train ...
Jazz's user avatar
  • 420
10 votes
Accepted

What's the difference of stateless LSTM and a normal feed-forward NN?

Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in ...
André Christoffer Andersen's user avatar
10 votes

What is the job of "RepeatVector" and "TimeDistributed"?

For encoder-decoder, your input is squashed into a single feature vector, if you want your output to regenerate the same dimension as the original input, you are "artificially" converting this feature ...
moon's user avatar
  • 201
9 votes
Accepted

Value Error: Operands could not be broadcast together with shapes - LSTM

The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light. First of all, you have to understand what is performed by the call ...
pcko1's user avatar
  • 3,950
9 votes

Advantages of stacking LSTMs?

From What are the advantages of stacking multiple LSTMs? (I'll only update the answer there): From {1}: While it is not theoretically clear what is the additional power gained by the ...
Franck Dernoncourt's user avatar
9 votes

Validation loss is not decreasing

Yes this is an overfitting problem since your curve shows point of inflection. This is a sign of very large number of epochs. In this case, model could be stopped at point of inflection or the number ...
Mohit Banerjee's user avatar
8 votes
Accepted

How to use Embedding() with 3D tensor in Keras?

I'm not entirely sure if this is the cleanest solution but I stitched everything together. Each of the 10 word positions get their own input but that shouldn't be too much of a problem. The idea is to ...
Jan van der Vegt's user avatar
8 votes

Are there any differences between Recurrent Neural Networks and Residual Neural Networks?

Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of ...
Green Falcon's user avatar
  • 14.1k

Only top scored, non community-wiki answers of a minimum length are eligible