Skip to main content

Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
125 votes
2 answers
116k views

Training an RNN with examples of different lengths in Keras

I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for ...
Tac-Tics's user avatar
  • 1,370
86 votes
8 answers
67k views

Time series prediction using ARIMA vs LSTM

The problem that I am dealing with is predicting time series values. I am looking at one time series at a time and based on for example 15% of the input data, I would like to predict its future values....
ahajib's user avatar
  • 1,075
59 votes
5 answers
79k views

Number of parameters in an LSTM model

How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the ...
wabbit's user avatar
  • 1,297
38 votes
4 answers
49k views

What is the meaning of "The number of units in the LSTM cell"?

From Tensorflow code: Tensorflow. RnnCell. num_units: int, The number of units in the LSTM cell. I can't understand what this means. What are the units of LSTM ...
Brans Ds's user avatar
  • 849
36 votes
1 answer
13k views

Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?

So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras. But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch ...
Rizky Luthfianto's user avatar
35 votes
6 answers
126k views

Validation loss is not decreasing

I am trying to train a LSTM model. Is this model suffering from overfitting? Here is train and validation loss graph:
DukeLover's user avatar
  • 601
24 votes
2 answers
23k views

What's the difference between the cell and hidden state in LSTM?

LSTM cells consist of two types of states, the cell state and hidden state. How do cell and hidden states differ, in terms of their functionality? What information do they carry?
user avatar
22 votes
1 answer
3k views

Understanding Timestamps and Batchsize of Keras LSTM considering Hiddenstates and TBPTT

What I'm trying to do What I am trying to do is predicting the next data-point $x_t$ for each point in the timeseries $[x_0, x_1, x_2,...,x_T]$ in the context of a date-stream in real-time, in theory ...
user avatar
21 votes
3 answers
56k views

What is LSTM, BiLSTM and when to use them?

I am very new to Deep learning and I am particularly interested in knowing what are LSTM and BiLSTM and when to use them (major application areas). Why are LSTM and BILSTM more popular than RNN? Can ...
Volka's user avatar
  • 711
19 votes
4 answers
17k views

What is the difference between word-based and char-based text generation RNNs?

While reading about text generation with Recurrent Neural Networks I noticed that some examples were implemented to generate text word by word and others character by character without actually ...
tastyminerals's user avatar
18 votes
3 answers
45k views

How to determine feature importance in a neural network?

I have a neural network to solve a time series forecasting problem. It is a sequence-to-sequence neural network and currently it is trained on samples each with ten features. The performance of the ...
Aesir's user avatar
  • 458
16 votes
1 answer
20k views

Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

I have been trying to understand how to represent and shape data to make a multidimentional and multivariate time series forecast using Keras (or TensorFlow) but I am still very unclear after reading ...
Bastien's user avatar
  • 263
16 votes
1 answer
10k views

RNN using multiple time series

I am trying to create a neural network using time series as input, in order to train it based on the type of each series. I read that using RNNs you can split the input into batches and use every ...
Ploo's user avatar
  • 333
15 votes
2 answers
29k views

Dropout on which layers of LSTM?

Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers? In Hinton's paper (which proposed ...
BigBadMe's user avatar
  • 750
15 votes
1 answer
12k views

Why do we need to add START <s> + END </s> symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to ...
alvas's user avatar
  • 2,410
15 votes
1 answer
27k views

How many LSTM cells should I use?

Are there any rules of thumb (or actual rules) pertaining to the minimum, maximum and "reasonable" amount of LSTM cells I should use? Specifically I am relating to BasicLSTMCell from TensorFlow and <...
user avatar
14 votes
1 answer
7k views

Forget Layer in a Recurrent Neural Network (RNN) -

I'm trying to figure out the dimensions of each variables in an RNN in the forget layer, however, I'm not sure if I'm on the right track. The next picture and equation is from Colah's blog post "...
user1157751's user avatar
14 votes
3 answers
33k views

what is darknet and why is it needed for YOLO object detection?

what is darknet and why is it needed for YOLO object detection ? I read that its a neural network written in C , but why is it needed for YOLO object detection when we have lot of machine learning ...
star's user avatar
  • 1,481
14 votes
2 answers
15k views

How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

I struggle to interpret the Keras coding difference for one-to-many (e. g. classification of single images) and many-to-many (e. g. classification of image sequences) sequence labeling. I frequently ...
Hendrik's user avatar
  • 8,667
13 votes
3 answers
2k views

An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

I would like to use ANNs for my problem, but the issue is my inputs and outputs node numbers are not fixed. I did some google searches before asking my question and found that the RNN may help me with ...
Vadim's user avatar
  • 303
13 votes
2 answers
23k views

When to use Stateful LSTM?

I'm trying to use LSTM on time-series data in order to generate future sequences that looks like the original sequences in term of values and progression direction. My approach is: train RNN to ...
Hastu's user avatar
  • 418
13 votes
1 answer
9k views

So what's the catch with LSTM?

I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying ...
I_Play_With_Data's user avatar
12 votes
5 answers
33k views

LSTM or other RNN package for R

I saw some impressive result from LSTM models producing Shakespeare like texts. I was wondering if an LSTM package exists for R. I googled for it but only found packages for Python and Julia. (maybe ...
Viktor's user avatar
  • 850
12 votes
1 answer
21k views

Keras LSTM with 1D time series

I'm learning how to use Keras and I've had reasonable success with my labelled dataset using the examples on Chollet's Deep Learning for Python. The data set is ~1000 Time Series with length 3125 with ...
user1147964's user avatar
12 votes
1 answer
417 views

ngram and RNN prediction rate wrt word index

I tried to plot the rate of correct predictions (for the top 1 shortlist) with relation to the word's position in sentence : I was expecting to see a plateau sooner on the ngram setup since it ...
Arkantus's user avatar
  • 157
11 votes
1 answer
6k views

What's the difference of stateless LSTM and a normal feed-forward NN?

From what I understand, the whole point of LSTM is for the network to establish long-term dependencies in the data, i.e. an event happening now may be in some way determined by something that happened ...
BigBadMe's user avatar
  • 750
11 votes
2 answers
2k views

How do attention mechanisms in RNNs learn weights for a variable length input

Attention mechanisms in RNNs are reasonably common to sequence to sequence models. I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output ...
davidparks21's user avatar
11 votes
2 answers
5k views

Trying to use TensorFlow to predict financial time series data

I'm new to ML and TensorFlow (I started about a few hours ago), and I'm trying to use it to predict the next few data points in a time series. I'm taking my input and doing this with it: ...
Isvara's user avatar
  • 211
11 votes
1 answer
2k views

Using RNN (LSTM) for Gesture Recognition System

I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it ...
Anas Ezz's user avatar
  • 111
11 votes
3 answers
1k views

Recurrent (CNN) model on EEG data

I'm wondering how to interpret a recurrent architecture in an EEG context. Specifically I'm thinking of this as a Recurrent CNN (as opposed to architectures like LSTM), but maybe it applies to other ...
Simon's user avatar
  • 1,071
10 votes
2 answers
5k views

How to train the same RNN over multiple series?

I have multiple separate time series and would like to train the same LSTM network on them. How to do in this situation? I can't just concatenate timeseries (along time), because I am afraid network ...
Dims's user avatar
  • 201
10 votes
1 answer
8k views

How to use Embedding() with 3D tensor in Keras?

I have a list of stock price sequences with 20 timesteps each. That's a 2D array of shape (total_seq, 20). I can reshape it into ...
offchan's user avatar
  • 305
10 votes
1 answer
605 views

Questions When Advancing from Vanilla Neural Network to Recurrent Neural Network

I've recently learned how a vanilla neural network would work, with given number of inputs, hidden nodes, and the same number of outputs as inputs. I've been looking at various posts now related to ...
Daniel's user avatar
  • 181
9 votes
2 answers
4k views

What's an LSTM-LM formulation?

I am reading this paper "Sequence to Sequence Learning with Neural Networks" http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf Under "2. The Model" it says: ...
Taivanbat Badamdorj's user avatar
9 votes
3 answers
319 views

Why do RNNs usually have fewer hidden layers than CNNs?

CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity. However, as far as I have seen, RNNs usually have few layers e.g. ...
KRL's user avatar
  • 231
9 votes
2 answers
1k views

Input for LSTM for financial time series directional prediction

I'm working on using an LSTM to predict the direction of the market for the next day. My question concerns the input for the LSTM. My data is a financial time series $x_1 \ldots x_t$ where each $x_i$...
articuno's user avatar
8 votes
3 answers
421 views

feature importance after classification

I have time series data and more or less 200 features for each sample, I used a recurrent neural network for the binary classification task. After the classification I would like to know which ...
Rick0's user avatar
  • 105
8 votes
3 answers
8k views

Clarification on the Keras Recurrent Unit Cell

I paste below the Keras documentation on Recurrent layer ...
Karthik 's user avatar
8 votes
1 answer
7k views

TensorFlow / Keras: What is stateful = True in LSTM layers?

Could you elaborate on this argument? I found the brief explanation from the docs unsatisfying: stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will ...
Leevo's user avatar
  • 6,265
8 votes
2 answers
8k views

Difference between explicit and implicit density with and without the relation to neural network

I have a lack of understanding about this issue. Could anybody explain it or give an advice to a good literature regarding it? I don't understand what is a explicit density model and how it differs ...
malocho's user avatar
  • 183
8 votes
2 answers
6k views

Why are RNN/LSTM preferred in time series analysis and not other NN?

I had recently a great discussion about the advantages of RNN/LSTM in time series analysis in comparison to other Neural Networks like MLP or CNN. The other side said, that: The NN just have to be ...
Mimi Müller's user avatar
8 votes
1 answer
8k views

Why doesn't training RNNs use 100% of the GPU?

I wonder why training RNNs typically doesn't use 100% of the GPU. For example, if I run this RNN benchmark on a Maxwell Titan X on Ubuntu 14.04.4 LTS x64, the GPU utilization is below 90%: The ...
Franck Dernoncourt's user avatar
8 votes
2 answers
2k views

Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros

I am a novice seeking help to gain experience in Data Science. Let us take a scenario where a big company would like to forecast its sales (a specific product) across different stores in different ...
wlaura's user avatar
  • 81
8 votes
1 answer
2k views

Recurrent neural network multiple types of input Keras

For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks....
Jan van der Vegt's user avatar
7 votes
2 answers
27k views

Number of parameters in an RNN

I'm using a basic RNN as in the figure below (say for translation). The model has the following structure: \begin{aligned} s_t &= \tanh(Ux_t + Ws_{t-1}) \\ o_t &= \mathrm{softmax}(Vs_t) \end{...
wabbit's user avatar
  • 1,297
7 votes
2 answers
7k views

Advantages of Recurrent Neural Networks over basic Artificial Neural Networks

I have started reading Deep Learning Book, and I am having trouble understanding the advantages of RNN. This part of confuses me: The unfodling process thus introduces two major advantages: ...
Stefan Radonjic's user avatar
7 votes
1 answer
2k views

Training with multi-series of different length with stateful LSTM

I'm training a stateful LSTM. My data is stored in a series of files, each file relates to a certain city. For each city I might have different amount of data, so City A I might have 4000 days, but ...
BigBadMe's user avatar
  • 750
7 votes
1 answer
357 views

A the end of a big DS project, should I make trained models available on GitHub?

I almost completed two big Data Science personal projects based on Deep Learning. They are the fanciest models I've implemented up to now, and I'm pushing all my code on GitHub. Do you advice to ...
Leevo's user avatar
  • 6,265
7 votes
1 answer
3k views

How to design batches in a stateful RNN

I am using TF Eager to train a stateful RNN (GRU). I have several variable length time sequences about 1 minute long which I split into windows of length 1s. In TF Eager, like in Keras, if ...
DankMasterDan's user avatar
7 votes
4 answers
19k views

How to evaluate performance of a time series model?

I trained a LSTM network on a time series dataset. Predictions seem to follow the dataset. In fact, they are nearly a right shifted form of real values. Thus, in my opinion, it doesn't provide any ...
Mustafa Orkun Acar's user avatar

1
2 3 4 5
15