Questions tagged [lstm]

LSTM stands for Long Short-Term Memory. When we use this term most of the time we refer to a recurrent neural network or a block (part) of a bigger network.

Filter by
Sorted by
Tagged with
156
votes
6answers
143k views

When to use GRU over LSTM?

The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates). Why do we make use of GRU ...
29
votes
1answer
10k views

Time Series prediction using LSTMs: Importance of making time series stationary

In this link on Stationarity and differencing, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, ...
26
votes
2answers
18k views

How to feed LSTM with different input array sizes?

If I like to write a LSTM network and feed it by different input array sizes, how is it possible? For example I want to get voice messages or text messages in a ...
23
votes
5answers
64k views

Validation loss is not decreasing

I am trying to train a LSTM model. Is this model suffering from overfitting? Here is train and validation loss graph:
20
votes
4answers
67k views

What does the output of model.predict function from Keras mean?

I have built a LSTM model to predict duplicate questions on the Quora official dataset. The test labels are 0 or 1. 1 indicates the question pair is duplicate. After building the model using ...
18
votes
2answers
12k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: <...
17
votes
4answers
14k views

Prediction interval around LSTM time series forecast

Is there a method to calculate the prediction interval (probability distribution) around a time series forecast from an LSTM (or other recurrent) neural network? Say, for example, I am predicting 10 ...
17
votes
3answers
12k views

Advantages of stacking LSTMs?

I'm wondering in what situations it is advantageous to stack LSTMs?
16
votes
1answer
19k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
16
votes
2answers
16k views

What is the job of "RepeatVector" and "TimeDistributed"?

I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing ...
15
votes
3answers
45k views

What is LSTM, BiLSTM and when to use them?

I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN? Can ...
15
votes
0answers
2k views

Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT

What I'm trying to do What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
14
votes
2answers
13k views

How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently ...
13
votes
2answers
16k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
13
votes
1answer
4k views

So what's the catch with LSTM?

I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying ...
12
votes
2answers
11k views

Batch Size of Stateful LSTM in keras

My Model is defined as below: ...
12
votes
2answers
16k views

When to use Stateful LSTM?

I'm trying to use LSTM on time-series data in order to generate future sequences that looks like the original sequences in term of values and progression direction. My approach is: train RNN to ...
11
votes
3answers
17k views

What is the best method for classification of time series data? Should I use LSTM or a different method?

I am trying to classify raw accelerometer data x,y,z to its corresponding label. What is the best architecture for best results? Or, does anyone have any suggestions on LSTM architectures built on ...
11
votes
2answers
13k views

Activation function between LSTM layers

I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation ...
11
votes
1answer
18k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
11
votes
1answer
2k views

Using RNN (LSTM) for Gesture Recognition System

I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it ...
10
votes
1answer
5k views

What's the difference of stateless LSTM and a normal feed-forward NN?

From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened ...
10
votes
1answer
7k views

How to use Embedding() with 3D tensor in Keras?

I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape (total_seq, 20). I can reshape it into ...
10
votes
3answers
8k views

Questions about LSTM cells, units and inputs

I'm trying to learn how LSTM networks work, and even if I get the basics, the details of the internal structure is not clear for me. On this blog link, I found ...
9
votes
2answers
4k views

How to train the same RNN over multiple series?

I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network ...
9
votes
4answers
12k views

How can I build a self-attention model with tf.keras.layers.Attention?

I have completed an easy many-to-one LSTM model as following. ...
9
votes
3answers
271 views

Why do RNNs usually have fewer hidden layers than CNNs?

CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity. However, as far as I have seen, RNNs usually have few layers e.g. ...
9
votes
2answers
3k views

LSTM Feature selection process

We need to implement a time series problem with the LSTM model. But, while implementing the same, the main challenge I am facing is the feature selection issue. Because our data-set contains 2300 ...
9
votes
1answer
689 views

How to arrange the dataset/images for CNN+LSTM

I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B). There is a time factor involved in this ...
9
votes
2answers
1k views

Input for LSTM for financial time series directional prediction

I'm working on using an LSTM to predict the direction of the market for the next day. My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$...
8
votes
2answers
6k views

Difference between LSTM cell state and hidden state

LSTM cells consist of two types of states, the cell state and hidden state. How do cell and hidden states differ, in terms of their functionality? What information do they carry?
8
votes
1answer
2k views

Binary classification of every time series step based on past and future values

I'm currently facing a Machine Learning problem and I've reached a point where I need some help to proceed. I have various time series of positional (x, ...
8
votes
2answers
4k views

LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process. ...
8
votes
2answers
2k views

Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros

I am a novice seeking help to gain experience in Data Science. Let us take a scenario where a big company would like to forecast its sales (a specific product) across different stores in different ...
7
votes
1answer
2k views

Training with multi-series of different length with stateful LSTM

I'm training a stateful LSTM. My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but ...
7
votes
1answer
7k views

How to get feature importance from a keras deep learning model?

In case of scikit-learn's models, we can get feature importance using the relevant attributes of the model. I've been working on a RNN, using LSTMs for text embedding. Is there any way to get ...
7
votes
1answer
1k views

what actually word embedding dimensions values represent?

I am learning word2vec and word embedding , I have downloaded GloVe pre-trained word embedding (shape 40,000 x 50) and using this function to extract information from that: ...
7
votes
1answer
3k views

How to draw a simple LSTM network

I'm new to deep learning, I am learning LSTM for my PhD work. This is a simple LSTM network for sequence classification. This code is from MATLAB tutorial: ...
7
votes
1answer
8k views

Multiple output for multi step ahead prediction using LSTM with keras

I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training ...
6
votes
2answers
24k views

Value Error: Operands could not be broadcast together with shapes - LSTM

I am trying an LSTM model using tensorflow following this tutorial . I am having trouble understanding why am I getting an ...
6
votes
2answers
2k views

Checking for stationarity in LSTM

An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked ...
6
votes
2answers
243 views

Why real-world output of my classifier has similar label ratio to training data?

I trained a neural network on balanced dataset, and it has good accuracy ~85%. But in real world positives appear in about 10% of the cases or less. When I test network on set with real world ...
6
votes
3answers
4k views

Multivariate Time series analysis: When is a CNN vs. LSTM appropriate?

I have multiple features in a time series and want to predict the values of the same features for the next time step. I have already trained an LSTM which is working okay, but takes a bit long to ...
6
votes
1answer
5k views

The model of LSTM with more than one unit

In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew ...
6
votes
1answer
6k views

Is it necessary to perform rolling-window on LSTMs?

Let's say I have a set of n time-series with sequence length 8 [[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]] And let's define the input that LSTM expects as a ...
6
votes
1answer
2k views

TensorFlow / Keras: What is stateful = True in LSTM layers?

Could you elaborate on this argument? I found the brief explanation from the docs unsatisfying: stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will ...
6
votes
1answer
663 views

Understanding LSTM structure

I am trying to learn LSTM's, and struggling a bit with the structure and the inputs/outputs of LSTM layers. Say I have a network definition like this: ...
6
votes
1answer
6k views

How to implement LSTM with Spark?

I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it. ...
6
votes
2answers
93 views

N-grams for RNNs

Given a word $w_{n}$ a statistical model such a Markov chain using n-grams predicts the subsequent word $w_{n+1}$. The prediction is by no means random. How is this translated into a neural model? I ...
6
votes
1answer
3k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...

1
2 3 4 5
21