Questions tagged [lstm]

LSTM stands for Long Short-Term Memory. When we use this term most of the time we refer to a recurrent neural network or a block (part) of a bigger network.

Filter by
Sorted by
Tagged with
105
votes
6answers
84k views

When to use GRU over LSTM?

The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates). Why do we make use of GRU ...
20
votes
1answer
5k views

Time Series prediction using LSTMs: Importance of making time series stationary

In this link on Stationarity and differencing, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, ...
15
votes
2answers
5k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: ...
14
votes
3answers
7k views

Advantages of stacking LSTMs?

I'm wondering in what situations it is advantageous to stack LSTMs?
13
votes
3answers
24k views

What does the output of model.predict function from Keras mean?

I have built a LSTM model to predict duplicate questions on the Quora official dataset. The test labels are 0 or 1. 1 indicates the question pair is duplicate. After building the model using ...
12
votes
1answer
15k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
12
votes
4answers
8k views

Prediction interval around LSTM time series forecast

Is there a method to calculate the prediction interval (probability distribution) around a time series forecast from an LSTM (or other recurrent) neural network? Say, for example, I am predicting 10 ...
12
votes
1answer
1k views

So what's the catch with LSTM?

I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying ...
12
votes
2answers
9k views

How to implement “one-to-many” and “many-to-many” sequence prediction in Keras?

I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently ...
11
votes
3answers
21k views

What is LSTM, BiLSTM and when to use them?

I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN? Can ...
10
votes
2answers
5k views

How to feed LSTM with different input array sizes?

If I like to write a LSTM network and feed it by different input array sizes, how is it possible? For example I want to get voice messages or text messages in a ...
10
votes
2answers
6k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
10
votes
1answer
15k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
10
votes
2answers
7k views

When to use Stateful LSTM?

I'm trying to use LSTM on time-series data in order to generate future sequences that looks like the original sequences in term of values and progression direction. My approach is: train RNN to ...
10
votes
1answer
1k views

Using RNN (LSTM) for Gesture Recognition System

I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it ...
8
votes
3answers
11k views

Validation loss is not decreasing

I am trying to train a LSTM model. Is this model suffering from overfitting? Here is train and validation loss graph:
8
votes
2answers
6k views

Batch Size of Stateful LSTM in keras

My Model is defined as below: ...
8
votes
1answer
5k views

How to use Embedding() with 3D tensor in Keras?

I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape (total_seq, 20). I can reshape it into (total_seq, 20, 1) for concatenation to other features. I also have ...
8
votes
3answers
189 views

Why do RNNs usually have fewer hidden layers than CNNs?

CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity. However, as far as I have seen, RNNs usually have few layers e.g. ...
8
votes
2answers
2k views

LSTM: How to deal with nonstationarity when predicting a time series

I want to do one-step-ahead predictions for time series with LSTM. To understand the algorithm, I built myself a toy example: A simple autocorrelated process. ...
7
votes
1answer
11k views

What is the best method for classification of time series data? Should I use LSTM or a different method?

I am trying to classify raw accelerometer data x,y,z to its corresponding label. What is the best architecture for best results? Or, does anyone have any suggestions on LSTM architectures built on ...
7
votes
2answers
2k views

How to train the same RNN over multiple series?

I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network ...
7
votes
3answers
4k views

Questions about LSTM cells, units and inputs

I'm trying to learn how LSTM networks work, and even if I get the basics, the details of the internal structure is not clear for me. On this blog link, I found ...
7
votes
1answer
5k views

Multiple output for multi step ahead prediction using LSTM with keras

I am new to deep learning and LSTM (with keras). I am trying to solve a multi-step ahead time series prediction. I have 3 time series: A, B and C and I want to predict the values of C. I am training ...
7
votes
0answers
253 views

How to arrange the dataset/images for CNN+LSTM

I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B). There is a time factor involved in this ...
7
votes
0answers
708 views

Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT

What I'm trying to do What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
6
votes
2answers
184 views

Why real-world output of my classifier has similar label ratio to training data?

I trained a neural network on balanced dataset, and it has good accuracy ~85%. But in real world positives appear in about 10% of the cases or less. When I test network on set with real world ...
6
votes
1answer
1k views

Training with multi-series of different length with stateful LSTM

I'm training a stateful LSTM. My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but ...
6
votes
1answer
1k views

Binary classification of every time series step based on past and future values

I'm currently facing a Machine Learning problem and I've reached a point where I need some help to proceed. I have various time series of positional (x, ...
6
votes
1answer
4k views

Is it necessary to perform rolling-window on LSTMs?

Let's say I have a set of n time-series with sequence length 8 [[a,b,c,d,e,f,g,h],[f,e,g,r,g,h,e,a],[a,e,r,a,k,e,l,i],...,[e,r,q,g,l,r,p,q]] And let's define the input that LSTM expects as a ...
6
votes
1answer
2k views

LSTM Feature selection process

We need to implement a time series problem by LSTM model. But, while implementing the same, the main challenge I am facing is feature selection issue. Because, our data-set contain 2300 observations ...
6
votes
1answer
100 views

Classifications as long-term memory and short-term memory in LSTM

How is the data classified as long-term memory and short-term memory? Is there some standards programmers set?
6
votes
1answer
646 views

Input for LSTM for financial time series directional prediction

I'm working on using an LSTM to predict the direction of the market for the next day. My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$...
5
votes
2answers
12k views

Value Error: Operands could not be broadcast together with shapes - LSTM

I am trying an LSTM model using tensorflow following this tutorial . I am having trouble understanding why am I getting an ...
5
votes
1answer
2k views

What's the difference of stateless LSTM and a normal feed-forward NN?

From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened ...
5
votes
2answers
942 views

Checking for stationarity in LSTM

An author in his blog checked for stationarity and removed them in a forecasting problem for using LSTM.I asked others and they said no need to check for it in LSTM.I read some articles and it looked ...
5
votes
2answers
4k views

What is the job of “RepeatVector” and “TimeDistributed”?

I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing ...
5
votes
2answers
2k views

Relationship between batch size and the number of neurons in the input layer

Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (...
5
votes
3answers
3k views

LSTM for time series - which window size to use

I have a LSTM based network which inputs a n-sized sequence of length (n x 300) and outputs the next single step (1 x 300). The "raw" data consists of a few thousand semi-processed sequences of ...
5
votes
1answer
40 views

Activation function between LSTM layers

I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation ...
5
votes
1answer
662 views

Why use two LSTM layers one after another?

In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example ...
5
votes
1answer
1k views

How to implement LSTM with Spark?

I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it. ...
5
votes
1answer
734 views

what actually word embedding dimensions values represent?

I am learning word2vec and word embedding , I have downloaded GloVe pre-trained word embedding (shape 40,000 x 50) and using this function to extract information from that: ...
5
votes
2answers
949 views

Anomaly detection using RNN LSTM

I'm trying to detect anomalies in an univariate time series. I trained a RNN LSTM and currently I get one-step-ahead predictions. Could someone explain if it's possible to output a confidence ...
5
votes
1answer
182 views

Suspected Exploding Gradient in Character Generator LSTM

I'm trying to create a neural network that can learn how to write text character by character from the book David Copperfield (via Project Gutenburg). It starts great, then forgets punctuation ...
5
votes
3answers
755 views

Kerns LSTM kernel

I am trying to understand how the weight matrix in an LSTM cell is used. An LSTM unit has several weight matrix: Wf, Wi, Wc, Wo like below: ( from http://colah....
4
votes
2answers
5k views

Time series forecasting with RNN(stateful LSTM) produces constant values

I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The ...
4
votes
3answers
1k views

Handwritting Recognition moving from character level to word level

Given the experience on MIST, I try this problem as a character level. I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, ...
4
votes
1answer
43 views

Do timesteps must have the same temporal distance in training a RNN?

I have a recurrent neural network with LSTM units that I want to train with batches of 6 timesteps. Each timestep is a record of a dataset and represents the temporal aggregation over 5 minutes of ...
4
votes
1answer
59 views

Why my results have time delay when I use LSTM?

I am trying to fit and test LSTM on a numeric series(like stock prices). But it seems that I always get a lag in predicted graph(Blue) with respect to real graph(red). Does anyone know why this ...