Questions tagged [lstm]

LSTM stands for Long Short-Term Memory. When we use this term most of the time we refer to a recurrent neural network or a block (part) of a bigger network.

Filter by
Sorted by
Tagged with
26
votes
2answers
18k views

How to feed LSTM with different input array sizes?

If I like to write a LSTM network and feed it by different input array sizes, how is it possible? For example I want to get voice messages or text messages in a ...
16
votes
1answer
19k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
11
votes
1answer
18k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
18
votes
2answers
12k views

Sliding window leads to overfitting in LSTM?

Will I overfit my LSTM if I train it via the sliding-window approach? Why do people not seem to use it for LSTMs? For a simplified example, assume that we have to predict the sequence of characters: <...
8
votes
2answers
6k views

Difference between LSTM cell state and hidden state

LSTM cells consist of two types of states, the cell state and hidden state. How do cell and hidden states differ, in terms of their functionality? What information do they carry?
3
votes
2answers
161 views

Time-series multi-step generalization from single step model

I have built a generic stacked lstm model of the form: ...
23
votes
5answers
64k views

Validation loss is not decreasing

I am trying to train a LSTM model. Is this model suffering from overfitting? Here is train and validation loss graph:
14
votes
2answers
13k views

How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently ...
11
votes
2answers
13k views

Activation function between LSTM layers

I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation ...
8
votes
1answer
2k views

Binary classification of every time series step based on past and future values

I'm currently facing a Machine Learning problem and I've reached a point where I need some help to proceed. I have various time series of positional (x, ...
4
votes
2answers
4k views

What is the vector value of [CLS] [SEP] tokens in BERT

In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector? I wanted to replace the proper nouns like ...
2
votes
2answers
4k views

loss/val_loss are decreasing but accuracies are the same in LSTM!

I am trying to train a LSTM model, but the problem is that the loss and val_loss are decreasing from 12 and 5 to less than 0.01, but the training set ...
29
votes
1answer
10k views

Time Series prediction using LSTMs: Importance of making time series stationary

In this link on Stationarity and differencing, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, ...
17
votes
4answers
14k views

Prediction interval around LSTM time series forecast

Is there a method to calculate the prediction interval (probability distribution) around a time series forecast from an LSTM (or other recurrent) neural network? Say, for example, I am predicting 10 ...
16
votes
2answers
16k views

What is the job of "RepeatVector" and "TimeDistributed"?

I read about them in Keras documentation and other websites, but I couldn't exactly understand what exactly they do and how should we use them in designing ...
11
votes
3answers
17k views

What is the best method for classification of time series data? Should I use LSTM or a different method?

I am trying to classify raw accelerometer data x,y,z to its corresponding label. What is the best architecture for best results? Or, does anyone have any suggestions on LSTM architectures built on ...
20
votes
4answers
68k views

What does the output of model.predict function from Keras mean?

I have built a LSTM model to predict duplicate questions on the Quora official dataset. The test labels are 0 or 1. 1 indicates the question pair is duplicate. After building the model using ...
6
votes
1answer
5k views

The model of LSTM with more than one unit

In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew ...
4
votes
1answer
5k views

Training an LSTM to track sine waves

I'm experimenting (read: playing around) with LSTMs on Keras. I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, ...
4
votes
1answer
932 views

RNNs for time series prediction - what configurations would make sense

My question here is mostly about general-intuition logic: when using a RNN (LSTM) for predicting a time series, and you have the goal of, for example, predicting at ...
12
votes
2answers
11k views

Batch Size of Stateful LSTM in keras

My Model is defined as below: ...
5
votes
2answers
7k views

Time series forecasting with RNN(stateful LSTM) produces constant values

I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The ...
5
votes
1answer
5k views

Is there an R tutorial of using LSTM for multivariate time series forecasting?

There is a great blog post about how to use keras stateful LSTM in R to forecast sunspots. I applied it to financial ts data ...
2
votes
0answers
948 views

Recommended model for univariate or multivariate multistep ahead time series forecasting

I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
2
votes
1answer
159 views

LSTM querying approach

I've just realized my prediction approach for LSTM might not be correct. I am trying to predict character by character, by reading over the book. The way I've approached the problem is as follows: <...
5
votes
2answers
6k views

Understanding input of LSTM

I am a little confused with the input of LSTM. Basicaly my train input data is of shape (53394, 3). I reshaped my 2D data into 3D data in order to set it according to the input of LSTM. I have two ...
5
votes
0answers
1k views

Multivariate, multistep forecasting with LSTM

I want to use an RNN with LSTM to forecast multiple steps into the future, based on multiple inputs. I have some ideas for different ways to approach this, but I'm afraid I'm missing the "right way" ...
3
votes
2answers
1k views

Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression?

Dear Data Science community, For a small project, I've started working on Neural networks as a regression tool, but I am still confused about possibilities of some variants. Here's what I am aiming ...
3
votes
2answers
140 views

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure: "the cat sat on the mat" -> [Seq2Seq ...
3
votes
0answers
208 views

Gradient derivation reference for Phased LSTM [closed]

I've worked-through the back-propagation for the Phased LSTM (Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu 2016) and would like to show the notes. It was a relatively difficult task, so I post it ...
3
votes
1answer
3k views

in TensorFlow 2.0, what is the different between LSTM and LSTMCell objects?

I am trying to implement an RNN in TensorFlow 2.0 (beta1). Looking at the layer functions (inherited from Keras) I found: tf.keras.layers.LSTM and ...
3
votes
1answer
1k views

Multivariate Time-Series forecasting using LSTM

I have a dataset of hourly measures of pollution('Sample_Measurement) and weather condition. If I want to predict the pollution level of the current hour using the weather and pollution data of the ...
2
votes
1answer
2k views

Shaping data for ConvLSTM for many-to-one image model

Ultimately, I am trying to obtain a binary segmentation mask for an image sequence. I have n number of image sequences, each with 500 greyscale images of size 256px by 400px. Each of these sequences ...
2
votes
1answer
2k views

LSTM Time series prediction for multiple multivariate series

I have to predict next min traffic for multiple cities (100+). I am thinking of using LSTM. My main concern is how do I scale the number of cities. How does LSTM learn different amount of traffic and ...
1
vote
1answer
3k views

How to design a many-to-many LSTM?

I have an input array of shape (1000,20, 4) and output(labels) of shape (1000,25,1). But don't know how to use Keras LSTM library to build a sequential model for this! Can someone help me design a ...
0
votes
1answer
495 views

LSTM Multi-class classification for large number of classes

I want to build a model that classifies 473 classes -product categories-, but I'm facing a problem with loss not decreasing. Data I have almost 3,000 data points for each class -473 classes- (data ...
5
votes
1answer
1k views

How to design batches in a stateful RNN

I am using TF Eager to train a stateful RNN (GRU). I have several variable length time sequences about 1 minute long which I split into windows of length 1s. In TF Eager, like in Keras, if ...
4
votes
1answer
130 views

Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data?

I'm training a deep network (CNN-LSTM-CRF) for Named Entity Recognition. Is there a reason that increasing the number of parameters would make the network less able to overfit a small training set (~...
3
votes
2answers
2k views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
3
votes
1answer
70 views

How to know if a time series sequence is predictibale or just random (Univariate time series prediction)?

I'm trying to predict a current value of a variable based on the its previous 10 values. I tried multiple time series approaches including ARIMA, LSTM and linear regression... None of them really ...
2
votes
1answer
704 views

How to optimally train deep learning model using output as new input

I'm trying to train a network to predict the future. My current setup uses 5 time steps as inputs from the past, each consisting of 10 features, resulting in a [5, 10] input matrix (initially ...
2
votes
1answer
555 views

Stateful LSTM : Using different training window

Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all ...
2
votes
2answers
2k views

Softmax classifier never allows for 100% probability in LSTM?

When working with LSTM I am using a softmax classifier and a one-hot encoded vector approach. The softmax looks like this: $$S(h_i) = \frac{e^{h_i}}{\sum e^{h_{total}}}$$ notice, LSTM's result is a $...
2
votes
1answer
3k views

How to implement LSTM using Doc2Vec vectors?

I would like to build a ANN for text classification, which has an LSTM layer, and using weights obtained via a Doc2Vec model ...
2
votes
1answer
138 views

Help framing a sequence prediction problem

I've found lots of tutorial/examples that focus on sequence prediction, which use previous time steps of the input variable(s) in order to create a forecast e.g. predict stock market price based on ...
2
votes
0answers
129 views

NER at sentence level or document level?

Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level? Let's say we have this input text, and we would like to do Named Entity Extraction: GOP Sen. Rand ...
2
votes
1answer
299 views

How to add previous predictions for new predictions in LSTM?

I am trying to train a model on a big data sequence like this [0.2 0.1 0.1 ..... 0.4 0.8] . I create X vectors with length 60 for inputs and Y scaler numbers as ...
2
votes
1answer
2k views

How to reshape data for LSTM training in multivariate sequence prediction

I want to build an LSTM model for customer behaviour. It's the first time for me working on a timeseries, so some concepts are not clear to me at all. My prediction problem is multidimensional, ...
2
votes
1answer
2k views

How to use TimeDistributed fo CNN+LSTM?

I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 ...
1
vote
0answers
49 views

Which One is the Best Way to Create Training Sequences for LSTM-based Class Prediction on Time-series Data?

Let's say I have time-series data in the following way. I need to create training sequences of a fixed length as an input to my LSTM model on PyTorch. ...