Questions tagged [lstm]

LSTM stands for Long Short-Term Memory. When we use this term most of the time we refer to a recurrent neural network or a block (part) of a bigger network.

Filter by
Sorted by
Tagged with
6
votes
2answers
123 views

N-grams for RNNs

Given a word $w_{n}$ a statistical model such a Markov chain using n-grams predicts the subsequent word $w_{n+1}$. The prediction is by no means random. How is this translated into a neural model? I ...
6
votes
1answer
3k views

What does SpatialDropout1D() do to output of Embedding() in Keras?

Keras model looks like this ...
6
votes
1answer
120 views

Classifications as long-term memory and short-term memory in LSTM

How is the data classified as long-term memory and short-term memory? Is there some standards programmers set?
6
votes
3answers
1k views

Kerns LSTM kernel

I am trying to understand how the weight matrix in an LSTM cell is used. An LSTM unit has several weight matrix: Wf, Wi, Wc, Wo like below: ( from http://colah....
6
votes
1answer
254 views

Training stateful LSTM with different number of sequences

I'm using a stateful LSTM for stock market analysis, and I have varying amounts of data for each stock, ranging from 20 years to just a few weeks (i.e. for newly listed stocks). I use 3 years of data ...
5
votes
2answers
6k views

Relationship between batch size and the number of neurons in the input layer

Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (...
5
votes
3answers
996 views

Deploying an LSTM Model

I have trained and validated my LSTM and I would like to deploy it. So, I know that we can save and load the Sequential object of Keras (I am working with Keras as you can guess). I thus implemented a ...
5
votes
3answers
8k views

LSTM for time series - which window size to use

I have a LSTM based network which inputs a n-sized sequence of length (n x 300) and outputs the next single step (1 x 300). The "raw" data consists of a few thousand semi-processed sequences of ...
5
votes
1answer
2k views

How to design batches in a stateful RNN

I am using TF Eager to train a stateful RNN (GRU). I have several variable length time sequences about 1 minute long which I split into windows of length 1s. In TF Eager, like in Keras, if ...
5
votes
2answers
8k views

Time series forecasting with RNN(stateful LSTM) produces constant values

I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The ...
5
votes
1answer
6k views

Is there an R tutorial of using LSTM for multivariate time series forecasting?

There is a great blog post about how to use keras stateful LSTM in R to forecast sunspots. I applied it to financial ts data ...
5
votes
1answer
1k views

Why use two LSTM layers one after another?

In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example ...
5
votes
2answers
7k views

Understanding input of LSTM

I am a little confused with the input of LSTM. Basicaly my train input data is of shape (53394, 3). I reshaped my 2D data into 3D data in order to set it according to the input of LSTM. I have two ...
5
votes
2answers
5k views

Initial embeddings for unknown, padding?

Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional <pad>, ...
5
votes
2answers
1k views

Anomaly detection using RNN LSTM

I'm trying to detect anomalies in an univariate time series. I trained a RNN LSTM and currently I get one-step-ahead predictions. Could someone explain if it's possible to output a confidence ...
5
votes
1answer
204 views

Suspected Exploding Gradient in Character Generator LSTM

I'm trying to create a neural network that can learn how to write text character by character from the book David Copperfield (via Project Gutenburg). It starts great, then forgets punctuation ...
5
votes
1answer
136 views

LSTM Long Term Dependencies Keras

I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras. In Keras the input data structure for X ...
5
votes
0answers
1k views

Multivariate, multistep forecasting with LSTM

I want to use an RNN with LSTM to forecast multiple steps into the future, based on multiple inputs. I have some ideas for different ways to approach this, but I'm afraid I'm missing the "right way" ...
4
votes
3answers
3k views

Are there any differences between Recurrent Neural Networks and Residual Neural Networks?

I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?
4
votes
3answers
370 views

Binary classification and numerical labels

I am trying to create a sentiment analysis model using a dataset that have ~50000 positive tweets that i labeled as 1, ~50000 negative tweets that i have labeled as 0. Also i have acquired ~10000 ...
4
votes
3answers
2k views

Why using a frozen embedding layer in an LSTM model

I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector ...
4
votes
1answer
1k views

Is a BiLSTM layer required if we use BERT?

I am new to Deep learning based NLP and I have a doubt - I am trying to build a NER model and I found some journals where people are relying on BERT-BiLSTM-CRF model for it. As far as I know BERT is a ...
4
votes
1answer
113 views

Which target variable should I use?

I have a problem where I want an LSTM to predict the resistance of a body. This value can also be calculated if we know the drag coefficient and the speed of that body. In my case, at inference time, ...
4
votes
2answers
4k views

What is the vector value of [CLS] [SEP] tokens in BERT

In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector? I wanted to replace the proper nouns like ...
4
votes
3answers
9k views

Keras' 'normal' LSTM uses the GPU?

I'm running Keras' LSTM (not CuDNNLSTM) but I notice my GPU is under load. I need recurrent dropout, so I can only stick with <...
4
votes
2answers
2k views

how to apply MC dropout to an LSTM network keras

I have a simple LSTM network developped using keras: ...
4
votes
2answers
1k views

Understanding LSTM input shape for keras

I am learning about the LSTM network. The input needs to be 3D. So I have a CSV file which has 9999 data with one feature only. So it is only one file. So usually it is ...
4
votes
1answer
5k views

Training an LSTM to track sine waves

I'm experimenting (read: playing around) with LSTMs on Keras. I want to train an LSTM network so it would "track" sine waves, that is, given sine waves with different wave length, phases and lengths, ...
4
votes
3answers
2k views

Handwritting Recognition moving from character level to word level

Given the experience on MIST, I try this problem as a character level. I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, ...
4
votes
2answers
156 views

Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure: "the cat sat on the mat" -> [Seq2Seq ...
4
votes
2answers
5k views

Forecasting via LSTM or XGBoost... is it really a forecast or

I guess I understand the idea of predictions made via LSTM or XGBoost models, but want to reach out to the community to confirm my thoughts. This tutorial does a nice job explaining step by step of ...
4
votes
1answer
123 views

Do timesteps must have the same temporal distance in training a RNN?

I have a recurrent neural network with LSTM units that I want to train with batches of 6 timesteps. Each timestep is a record of a dataset and represents the temporal aggregation over 5 minutes of ...
4
votes
1answer
1k views

Fixed effects or random effects in RNN

Lately, I have been concerned to implement fixed effects and random effects (from econometrics) in deep learning. After reading some articles, I realized that most of them just used only the neural ...
4
votes
1answer
131 views

Why my results have time delay when I use LSTM?

I am trying to fit and test LSTM on a numeric series(like stock prices). But it seems that I always get a lag in predicted graph(Blue) with respect to real graph(red). Does anyone know why this ...
4
votes
2answers
422 views

Multiple-output vs single-output NNs

I'm trying to build a 5 input-5 output model using LSTM, where all the outputs are the same features as the inputs, predicted in the future. My question is: is it better to build 5 models, each with ...
4
votes
1answer
981 views

RNNs for time series prediction - what configurations would make sense

My question here is mostly about general-intuition logic: when using a RNN (LSTM) for predicting a time series, and you have the goal of, for example, predicting at ...
4
votes
2answers
936 views

RNN in pseudo-code

A few years ago, I understood the classical MLP neural network much better when I wrote an implementation from scratch (using only Python + Numpy, without using tensorflow). Now I'd like to do the ...
4
votes
2answers
3k views

Can I use LSTM models to evaluate multiple, independent time series?

Let's say that I would like to predict the temperature tomorrow. I could use the approach whereby I train a model based on a time-series dataset collected from a single location (for example, see this ...
4
votes
2answers
6k views

Generate new sentences based on keywords

For example, for a domain specific neural network in Fashion, with the Keywords light, dress, orange, cotton. It could output: This gorgeous orange summer dress is great for wearing on sunny camping ...
4
votes
2answers
1k views

Trying to understand encoder-decoder sequential models in Keras?

My understanding is that for some types of seq2seq models, you train an encoder and a decoder, and then you set aside the encoder and use only the decoder for the prediction step. For example this ...
4
votes
1answer
132 views

How to predict value in every 120 minutes using LSTM in python

I want to predict value in every 120 minutes continuous using LSTM model. Here I wrote the code for predction. But I'm not getting proper prediction values . Here from start time I need to predict ...
4
votes
1answer
4k views

In Keras, how to get 3D input and 3D output for LSTM layers

In my original setting, I got X1 = (1200,40,1) y1 = (1200,10) Then, I work perfectly with my codes: ...
4
votes
1answer
2k views

How to recognise when to stop training based on Overfitting/Underfitting?

I am trying to train a LSTM network, over a total of 200 epochs, with hidden layer size of 100 and 1 dense layer after the LSTM layer. I have used a batch size of 10 for the same. Basically, I am ...
4
votes
1answer
250 views

On the choice of LSTM input/output dimension for a spatio-temporal problem

I am using LSTM neural networks from (R)Keras for a matter of spatio-temporal interpolation. I manage to get the network to output predictions but the results are not outstanding (very little ...
4
votes
3answers
1k views

What is the difference between "Adding more LSTM layers" or "Adding more units on existence layers"?

What is the difference between adding more LSTM layers and just increasing the units of existing layers? Which one is preferred and in which situation?
4
votes
2answers
1k views

Is this a problem for a Seq2Seq model?

I'm struggling to find a tutorial/example which covers using an seq2seq model for sequential inputs other then text/translation. I have a multivariate dataset with n number of input variables each ...
4
votes
1answer
132 views

Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data?

I'm training a deep network (CNN-LSTM-CRF) for Named Entity Recognition. Is there a reason that increasing the number of parameters would make the network less able to overfit a small training set (~...
4
votes
1answer
3k views

Kalman filter for time series prediction

I have the information about the behaviour of 400 users across period of 1 months (30 days). Across those 30 days I measure 4 different information (let's call it A,B,C and D), hence I have a total of ...
4
votes
1answer
3k views

Autoencoders for the compression of time series

I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series. Here are the models I tried. Simple autoencoder: ...
4
votes
1answer
919 views

Train LSTM model with multiple time series

I am predicting energy usage for a bedroom within a school residential building with date, temperature, and humidity as input features, using 7 time-steps and ...

1
2
3 4 5
21