116
votes
When to use GRU over LSTM?
GRUs and LSTMs utilize different approaches toward gating information to prevent the vanishing gradient problem. Here are the main points comparing the two:
The GRU unit controls the flow of ...
- 1,929
72
votes
When to use GRU over LSTM?
*To complement already great answers above.
From my experience, GRUs train faster and perform better than LSTMs on less training data if you are doing language modeling (not sure about other tasks).
...
- 2,117
44
votes
Accepted
Time Series prediction using LSTMs: Importance of making time series stationary
In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training ...
- 2,178
44
votes
Accepted
How to feed LSTM with different input array sizes?
The easiest way is to use Padding and Masking.
There are three general ways to handle variable-length sequences:
Padding and masking (which can be used for (3)),
Batch size = 1, and
Batch size > 1,...
- 8,967
36
votes
Validation loss is not decreasing
The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing.
Dealing with such a Model:
Data Preprocessing: Standardizing and Normalizing the ...
- 625
23
votes
When to use GRU over LSTM?
FULL GRU Unit
$ \tilde{c}_t = \tanh(W_c [G_r * c_{t-1}, x_t ] + b_c) $
$ G_u = \sigma(W_u [ c_{t-1}, x_t ] + b_u) $
$ G_r = \sigma(W_r [ c_{t-1}, x_t ] + b_r) $
$ c_t = G_u * \tilde{c}_t + (1 - ...
- 341
16
votes
Accepted
Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras
The input shape for an LSTM must be (num_samples, num_time_steps, num_features). In your example case, combining both cities as input, ...
- 14.4k
16
votes
Dropout on which layers of LSTM?
I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important ...
- 13.6k
16
votes
What's the difference between the cell and hidden state in LSTM?
In short:
Cell state: Long term memory of the model, only part of LSTM models
Hidden state: Working memory, part of LSTM and RNN models
Additional Information
RNN and vanishing/exploding gradients
...
- 1,918
15
votes
When to use GRU over LSTM?
This answer actually lies on the dataset and the use case. It's hard to tell definitively which is better.
GRU exposes the complete memory unlike LSTM, so applications which
that acts as advantage ...
- 2,296
15
votes
Accepted
What is LSTM, BiLSTM and when to use them?
RNN architectures like LSTM and BiLSTM are used in occasions where the learning problem is ...
- 13.6k
13
votes
Accepted
Sliding window leads to overfitting in LSTM?
Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for ...
- 1,971
13
votes
What is the best method for classification of time series data? Should I use LSTM or a different method?
I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies ...
- 3,870
13
votes
Accepted
Activation function between LSTM layers
Simple explanation with images
We know that an activation is required between matrix multiplications to afford a neural network the ability to model non-linear processes.
A classical LSTM cell ...
- 14.4k
12
votes
Accepted
Prediction interval around LSTM time series forecast
Directly, this is not possible. However, if you model it in a different way you can get out confidence intervals. You could instead of a normal regression approach it as estimating a continuous ...
- 9,248
12
votes
So what's the catch with LSTM?
You are right that LSTMs work very well for some problems, but some of the drawbacks are:
LSTMs take longer to train
LSTMs require more memory to train
LSTMs are easy to overfit
Dropout is much ...
- 2,341
11
votes
Accepted
Keras LSTM with 1D time series
LSTM layers require data of a different shape.
From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should ...
- 14.4k
11
votes
What does the output of model.predict function from Keras mean?
The output of a neural network will never, by default, be binary - i.e. zeros or ones. The network works with continuous values (not discrete) in order to optimise the loss more freely in the ...
- 14.4k
10
votes
Accepted
Checking for stationarity in LSTM
In principle no you do not need to check for stationarity nor correct for it when you are using an LSTM.
The thing about stationarity is that it makes prediction tasks much more efficient, and stable....
- 702
10
votes
Accepted
Batch Size of Stateful LSTM in keras
I solved the problem this way:
I realized that I needed to find the HCF (highest common factor) of both the length of x_train ...
- 421
10
votes
Accepted
What's the difference of stateless LSTM and a normal feed-forward NN?
Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in ...
10
votes
Accepted
What is the job of "RepeatVector" and "TimeDistributed"?
tf.keras.layers.RepeatVector
According to the docs :
Repeats the input n times.
They have also provided an example :
...
- 2,055
9
votes
Accepted
Value Error: Operands could not be broadcast together with shapes - LSTM
The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.
First of all, you have to understand what is performed by the call
...
- 3,870
9
votes
When to use Stateful LSTM?
As for stateful LSTM and its understanding, refer to here. Quoting an answer from there:
"I’m given a big sequence (e.g. Time Series) and I split it into smaller sequences to construct my input ...
- 3,870
9
votes
Binary classification of every time series step based on past and future values
You are facing a very common problem: handling imbalanced data. For neural networks, typical procedures are:
Having the proper metrics: global accuracy should not be used.
Oversampling the minority ...
- 1,618
9
votes
What is the job of "RepeatVector" and "TimeDistributed"?
For encoder-decoder, your input is squashed into a single feature vector, if you want your output to regenerate the same dimension as the original input, you are "artificially" converting this feature ...
- 191
8
votes
Accepted
How to use Embedding() with 3D tensor in Keras?
I'm not entirely sure if this is the cleanest solution but I stitched everything together. Each of the 10 word positions get their own input but that shouldn't be too much of a problem. The idea is to ...
- 9,248
8
votes
Advantages of stacking LSTMs?
From What are the advantages of stacking multiple LSTMs? (I'll only update the answer there):
From {1}:
While it is not theoretically clear what is the additional power gained by the ...
- 5,478
8
votes
Are there any differences between Recurrent Neural Networks and Residual Neural Networks?
Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of ...
- 13.6k
8
votes
Stock prediction through LSTM
Yes, this is overfitting. Financial time series exhibit many peculiarities, including heteroscedasticity, high tail risk, all kinds of seasonality and all kinds of momentum. Machine learning ...
- 1,636
Only top scored, non community-wiki answers of a minimum length are eligible