Skip to main content

Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
16 votes
1 answer
20k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
Bastien's user avatar
  • 263
125 votes
2 answers
116k views

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
Tac-Tics's user avatar
  • 1,370
59 votes
5 answers
79k views

Number of parameters in an LSTM model

How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the ...
wabbit's user avatar
  • 1,297
12 votes
1 answer
21k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
user1147964's user avatar
35 votes
6 answers
126k views

Validation loss is not decreasing

I am trying to train a LSTM model. Is this model suffering from overfitting? Here is train and validation loss graph:
DukeLover's user avatar
  • 601
24 votes
2 answers
23k views

What's the difference between the cell and hidden state in LSTM?

LSTM cells consist of two types of states, the cell state and hidden state. How do cell and hidden states differ, in terms of their functionality? What information do they carry?
user avatar
86 votes
8 answers
67k views

Time series prediction using ARIMA vs LSTM

The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values....
ahajib's user avatar
  • 1,075
18 votes
3 answers
45k views

How to determine feature importance in a neural network?

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
Aesir's user avatar
  • 458
14 votes
1 answer
7k views

Forget Layer in a Recurrent Neural Network (RNN) -

I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from Colah's blog post "...
user1157751's user avatar
14 votes
2 answers
15k views

How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently ...
Hendrik's user avatar
  • 8,667
8 votes
1 answer
7k views

TensorFlow / Keras: What is stateful = True in LSTM layers?

Could you elaborate on this argument? I found the brief explanation from the docs unsatisfying: stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will ...
Leevo's user avatar
  • 6,265
6 votes
2 answers
5k views

How to sort numbers using Convolutional Neural Network?

Recently, in an interview I got this question: Design a convnet that sorts numbers. Operators are ReLU, Conv, and Pooling. E.g. input: 5, 3, 6, 2; output: 2, 3, 5, 6 I am not sure how can you sort a ...
rise of a phoenix's user avatar
6 votes
2 answers
17k views

Loss being outputed as nan in keras RNN

Since the first Epoch of the RNN, the loss value is being outputted as nan. Epoch 1/100 9787/9787 [==============================] - 22s 2ms/step - loss: nan I have normalized the data. ...
Erik Dz's user avatar
  • 73
6 votes
1 answer
7k views

The model of LSTM with more than one unit

In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew ...
PoCheng.Lin's user avatar
5 votes
2 answers
7k views

RNN time-series predictions with multiple features containing non-numeric features and numeric features?

The question RNN's with multiple features is ambiguous and not explicitly in differentiating different features. I want to understand how to use RNN to predict time-series with multiple features ...
hhh's user avatar
  • 171
4 votes
1 answer
3k views

Padding sequences for neural sequence models (RNNs)

I am padding sequences for a GRU based classifier that I am building in Keras. I'm wondering if there's any accepted best practice for padding the leading or trailing side of the sequence. E.g. <...
Sledge's user avatar
  • 254
15 votes
2 answers
29k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
BigBadMe's user avatar
  • 750
15 votes
1 answer
12k views

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to ...
alvas's user avatar
  • 2,410
13 votes
1 answer
9k views

So what's the catch with LSTM?

I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying ...
I_Play_With_Data's user avatar
7 votes
2 answers
7k views

Advantages of Recurrent Neural Networks over basic Artificial Neural Networks

I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me: The unfodling process thus introduces two major advantages: ...
Stefan Radonjic's user avatar
7 votes
0 answers
1k views

Multivariate, multistep forecasting with LSTM

I want to use an RNN with LSTM to forecast multiple steps into the future, based on multiple inputs. I have some ideas for different ways to approach this, but I'm afraid I'm missing the "right way" ...
Aurast's user avatar
  • 171
7 votes
2 answers
27k views

Number of parameters in an RNN

I'm using a basic RNN as in the figure below (say for translation). The model has the following structure: \begin{aligned} s_t &= \tanh(Ux_t + Ws_{t-1}) \\ o_t &= \mathrm{softmax}(Vs_t) \end{...
wabbit's user avatar
  • 1,297
7 votes
1 answer
3k views

How to design batches in a stateful RNN

I am using TF Eager to train a stateful RNN (GRU). I have several variable length time sequences about 1 minute long which I split into windows of length 1s. In TF Eager, like in Keras, if ...
DankMasterDan's user avatar
7 votes
1 answer
4k views

Need help understanding LSTMs' backpropagation and carousel of error

I'm having a hard time trying to derive the maths behind LSTMs and vanishing gradients. I had a of help from LSTM forward and backward pass, but I got stuck in page 11 from LSTM forward and backward ...
user1157751's user avatar
5 votes
2 answers
4k views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
Osama Dar's user avatar
  • 599
5 votes
2 answers
8k views

Text similarity using RNN

Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the ...
user28251's user avatar
4 votes
1 answer
1k views

Train LSTM model with multiple time series

I am predicting energy usage for a bedroom within a school residential building with date, temperature, and humidity as input features, using 7 time-steps and ...
sunday's user avatar
  • 41
4 votes
1 answer
1k views

RNNs for time series prediction - what configurations would make sense

My question here is mostly about general-intuition logic: when using a RNN (LSTM) for predicting a time series, and you have the goal of, for example, predicting at ...
NeuronQ's user avatar
  • 93
3 votes
1 answer
3k views

in TensorFlow 2.0, what is the different between LSTM and LSTMCell objects?

I am trying to implement an RNN in TensorFlow 2.0 (beta1). Looking at the layer functions (inherited from Keras) I found: tf.keras.layers.LSTM and ...
Leevo's user avatar
  • 6,265
2 votes
0 answers
1k views

Recommended model for univariate or multivariate multistep ahead time series forecasting

I have a dataset consisting of recurring and non-recurring expense transactions from bank accounts, as well as other features describing the bank account and each transation. I aggregate these ...
KOB's user avatar
  • 189
2 votes
2 answers
2k views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
Leevo's user avatar
  • 6,265
2 votes
1 answer
188 views

Help framing a sequence prediction problem

I've found lots of tutorial/examples that focus on sequence prediction, which use previous time steps of the input variable(s) in order to create a forecast e.g. predict stock market price based on ...
Ellio's user avatar
  • 93
2 votes
1 answer
676 views

Stateful LSTM : Using different training window

Would it make sense for stateful LSTM (or LSTM in general) if in one epoch I feed [0-9],[10-19],[20-29],[30-39]...[990-999] (with corresponding labels/Y data) from my dataset. When I've presented all ...
BigBadMe's user avatar
  • 750
2 votes
1 answer
1k views

How to feed a table per timestamp to LSTM neural network?

I have a time-series dataframe like this ...
Extermis's user avatar
2 votes
1 answer
67 views

Why RNNs necessary for time series?

I got it that when using time series data, I have to use a RNN The highlight here is that the neurons receive their output again as an input, so they can take into account the previous step. But ...
Ben's user avatar
  • 560
2 votes
1 answer
2k views

How to reshape data for LSTM training in multivariate sequence prediction

I want to build an LSTM model for customer behaviour. It's the first time for me working on a timeseries, so some concepts are not clear to me at all. My prediction problem is multidimensional, ...
ginevracoal's user avatar
1 vote
2 answers
90 views

Can I create a layer with multiple rnn cell ? [question about a paper]

I am trying to implement https://dl.acm.org/doi/pdf/10.1145/3269206.3271794 . Structure: As it said: In particular, we integrate the embedding vectors learned from each individual recurrent encoder ...
Mithril's user avatar
  • 383
1 vote
2 answers
2k views

RNN with PyTorch - I don't understand the initial parameters

I would like to understand the pyTorch RNN module in detail. There I created a very simple and basic example: ...
Thomas K's user avatar
1 vote
1 answer
222 views

Sentiment analysis for multiple entry in one text

I would like to do sentiment analysis on a set of financial news from the S&P 500 for given entities (organization names). However, each news (rows in my dataset) may have more than one entity and ...
M. Ebrahimi's user avatar
1 vote
2 answers
3k views

Multivariate and multi-series LSTM

I am trying to create a pollution prediction LSTM. I've seen an example on the web to cater for a Multivariate LSTM to predict the pollution levels for one city (Beijing), but what about more than one ...
BigBadMe's user avatar
  • 750
1 vote
0 answers
1k views

RNN for classification giving vastly different results (Keras)

I have a time series data (counts of the number of infectious individuals per day), and I am using a SimpleRNN from Keras to classify each of these epidemics into one of eight possible classes. I ...
StatsSorceress's user avatar
0 votes
2 answers
157 views

Keras: ambiguity regarding state maintenance in RNNs

The following is mentioned in the official keras RNN documentation (https://www.tensorflow.org/guide/keras/rnn). By "Normally", I assume they mean when stateful=False, which is also the ...
Enk9456's user avatar
  • 105
0 votes
1 answer
388 views

About batches in stateful RNN

..., to create proper consecutive batches, where the nth input sequence in a batch starts off exactly where the nth input sequence ended in the previous batch. Géron, Aurélien. Hands-On Machine ...
Crispy13's user avatar
  • 133
0 votes
1 answer
135 views

How can I picture an unfolded RNN as a normal Feed Forward Network?

I am currently working on a Transformer architecture. Trying to picture an RNN (or Encoder) as a normal Feed Forward network really confused me after looking at the following image in an article: (...
Fishie's user avatar
  • 1