Skip to main content

Questions tagged [rnn]

A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed cycle.

Filter by
Sorted by
Tagged with
3 votes
1 answer
97 views

Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

I understand the concept of backpropagation in standard neural networks and backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of ...
jim22394's user avatar
0 votes
0 answers
60 views

Trying to extend this code to include additional feature volume (in addition to adj close) RNN to predict adj close

I read this article on medium https://medium.com/swlh/a-technical-guide-on-rnn-lstm-gru-for-stock-price-prediction-bce2f7f30346 prep ...
thistleknot's user avatar
0 votes
1 answer
703 views

How to improve LSTM accuracy on multiclass text classification?

So, I'm trying to build a LSTM model to classify multiclass text label. The goal is to make a prediction about user rating (1, 2, 3, 4, 5) based on their review. My hyperparameter is like this: ...
Jericho's user avatar
2 votes
1 answer
268 views

Can Batch Normalization replace tanh in RNN?

Question Can Batch Normalization (BN) be inserted in RNN after $x_t@W_{xh}$, and after $h_{t-1}@W_{hh}$ to remove $f=tanh$ and bias $b_h$? If possible, will this ...
mon's user avatar
  • 731
2 votes
0 answers
473 views

Elman RNN with keras

I have to perform multi-step multivariate forecasting of time series, using keras. I found an example where LSTM is used. I could modify that example replacing LSTM with SimpleRNN. Now I would like to ...
Oldville's user avatar
  • 141
1 vote
0 answers
158 views

Why don't we set initial hidden states in RNN to random small numbers like we do to the weights?

I'm following a couple of tutorials on RNNs and the instructor said that we should always set the initial hidden state in our RNNs to a tensor of all zeros and I couldn't really understand why. Even ...
tildawn's user avatar
  • 11
3 votes
2 answers
255 views

Do Recurrent Neural Networks assume stationarity or just a general kind of sequential dependence?

Just when I thought I had convinced myself that RNNs make no other assumption about a sequence other than that there are dependencies between the inputs and that (in the case of monodirectional RNNs) ...
Tom's user avatar
  • 43
1 vote
1 answer
65 views

How can I convert my predictions to text after predicting using RNN?

I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different. ...
Igbal's user avatar
  • 11
1 vote
0 answers
190 views

Padding before or after truncation for stateful LSTM in Keras

I am training an LSTM on a dataset with variable timesteps (between 10 and 6000). Using the truncated backpropagation through time (TBPTT) technique, I am truncating the sequences to windows of 128 ...
ProteinGuy's user avatar
-1 votes
2 answers
37 views

Optimisation of neural networks

Do neural networks get optimized by trial and error, by data scientists, or is there some way of optimizing values through accurate mathematical equations?
Domenico Bagnato's user avatar
0 votes
1 answer
380 views

Tricky stacking models in keras

I'm trying to write a model with keras, that is built as shown below: ...
Georgy Firsov's user avatar
0 votes
1 answer
1k views

What is the input of LSTM network?

Hello I am trying to understand LSTMs but have a few problems: What is the input? Since LSTM is seq2seq I would think it is a sequence of words, but in a Codecademy lesson is mentioned that each ...
Tknoobs's user avatar
  • 15
1 vote
0 answers
12 views

why would you mask out padded activations from the training loss?

I've followed taming-lstm for training a LSTM model on a NLP task in batches with various sentence lengths. One of his main points is: Trick 3: Mask out network outputs we don’t want to consider in ...
ihadanny's user avatar
  • 1,357
0 votes
2 answers
657 views

Difference between zero-padding and character-padding in Recurrent Neural Networks

For RNN's to work efficiently we vectorize the problem which results in an input matrix of shape (m, max_seq_len) where m is the number of examples, e.g. ...
PhysicsMan's user avatar
2 votes
2 answers
602 views

Timeseries LSTM: does test data need to come after training data?

I have one single, very long time series. I want to train an LSTM to distinguish between two behaviours (A or B) at every timestep (sequence-to-sequence). Because the time series is very long, I plan ...
lodo's user avatar
  • 121
0 votes
0 answers
158 views

Making Keras RNN to proceed input sequence step by step

I'm currently trying to create a neural network for playing Tetris. I'm using evolutionary algorythms for it's learning, so the behavior that I need to get from the neural network is the following: ...
Nick Zorander's user avatar
0 votes
1 answer
546 views

Regression with LSTM network: use multiple time series as input

I've spent a few days on this and am starting to think I'm missing the obvious solution as this doesn't seem like a very uncommon problem. As an example dataset: I have 100 measurements with each a ...
Emyn Arnen's user avatar
1 vote
0 answers
361 views

How to implement a Multivariate multi-site application in LSTM?

I am trying to make a multivariate multi-site classification LSTM model using Keras. I have followed this tutorial from Jason Brownlee: https://machinelearningmastery.com/multivariate-time-series-...
Ralph's user avatar
  • 11
0 votes
0 answers
249 views

How to improve a LSTM self-attention model given the absence of overfitting

I am doing a binary classification on time series data. Class 0 is a single class but class 1 is actually a combination of 7 different classes. My objective to classify class 0 from other classes. The ...
Leo's user avatar
  • 125
1 vote
1 answer
1k views

Number of parameters in Simple RNNs

Please, I am stuck, I can not understand the number of parameters of a simple RNN, here the example and the model summary. the example is simple: ...
Abir ELTAIEF's user avatar
1 vote
0 answers
64 views

Advice on using Recurrent Neural Networks for non-time series dataset

I'm testing different machine learning algorithms for predicting week-to-week fantasy football scores for individual players. For those who don't know, fantasy football is a game in which players pick ...
Heathcliff's user avatar
1 vote
0 answers
238 views

Concatenation of CNN and LSTM to model time of a series of images

I have collected a dataset consisting of around 30'000 heat maps of 80 users. The heat maps represent typing behavior on a keyboard and are just images with a resolution of ...
machinery's user avatar
  • 246
0 votes
1 answer
639 views

Understanding projection layer for BLSTM

In many research papers there are 'projection layers' related to BLSTM layers. For example, from here: "we trained an 8-layer BLSTM encoder including 320 cells in each layer and direction, and ...
Selma_KA's user avatar
1 vote
1 answer
47 views

Long range forecasting with sequence-to-sequence models

I have a task where I want to forecast daily observations for 1 year or 2 years in advance at multiple locations--so 365 or 730 days in advance. I actually have a pretty good dataset, meaning daily ...
krishnab's user avatar
  • 163
2 votes
2 answers
346 views

Understanding of number of cells in layers of sequential models

I am trying to teach myself RNN, but I have a question. And so, imagine 2 layers: an input layer with three neurons $(x1, x2, x3)$ and a classic recurrent layer with 2 neurons and an activation ...
Nikto's user avatar
  • 131
2 votes
1 answer
57 views

Modeling Encoder-Decoder according to instructions from a paper

I am new to this field and I was reading a paper "Predicting citation counts based on deep neural network learning techniques". There the authors describe the code that they implemented if ...
oaksandbrooms's user avatar
0 votes
1 answer
121 views

Optimal input setup for character-level text classification RNN

I want to classify 500-character long text samples as to whether they look like natural language using a character-level RNN. I'm unsure as to the best way to feed the input to the RNN. Here are two ...
Dave White's user avatar
0 votes
1 answer
319 views

Are there any control-flow/conditional statements in AI/ML models?

I was recently asked this during an interview. When we write a C program, it has a control-flow in the form of conditional statements like if, ...
Joe Black's user avatar
  • 123
1 vote
2 answers
90 views

Can I create a layer with multiple rnn cell ? [question about a paper]

I am trying to implement https://dl.acm.org/doi/pdf/10.1145/3269206.3271794 . Structure: As it said: In particular, we integrate the embedding vectors learned from each individual recurrent encoder ...
Mithril's user avatar
  • 383
1 vote
2 answers
64 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
user2351494's user avatar
0 votes
1 answer
491 views

How is the hidden state of a GRU initialized

This is a GRU. Now, what will be the value of $h_t$, at $t$=$0$. That is, what will be the value of the hidden state at just the starting?
Batman's user avatar
  • 1
6 votes
2 answers
673 views

How is it possible for RNN to do sentiment analysis?

I'm wondering how RNN can be used when doing sentiment analysis. It seems that the characteristic of RNN is to remember what appeared in the past and determine the value of the present (future), but I ...
WooseokChoi's user avatar
1 vote
0 answers
611 views

Implementation of the LSTM using Keras in R with multiple outputs

I'm implementing the LSTM based on this tutorial (https://blogs.rstudio.com/ai/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/), but the example consider multiple inputs with ...
Fernanda's user avatar
1 vote
0 answers
127 views

Machine learning for circular sequences

My data are sequences of real numbers $a_0,a_1,...,a_{n-1}$. The length of a sequence is fixed and equals $n$. Each sequence is mapped to a real number $y$ and I want to predict $y$ given the sequence....
Vladislav Gladkikh's user avatar
3 votes
0 answers
81 views

How is the input gate in the LSTM learn?

How is the input gate neural network trained what to remember by propagating the error rate from predicting the next word in the language model? How does it help it to learn if it remembered the right ...
Drxxd's user avatar
  • 131
24 votes
2 answers
23k views

What's the difference between the cell and hidden state in LSTM?

LSTM cells consist of two types of states, the cell state and hidden state. How do cell and hidden states differ, in terms of their functionality? What information do they carry?
user avatar
1 vote
1 answer
3k views

what is the complexity of a bidirectional recurrent neural network?

In particular, what is the complexity of a bi-directional recurrent neural network taking into account the variants of LSTM and GRU as well for training? I am hoping if I can get links to some ...
Sourajit Behera's user avatar
1 vote
1 answer
793 views

LSTM Target Is Also One of It's Inputs?

I have two input arrays that include both historical and forecasted data, and one input array that is only historical. I'm trying to predict (or "forecast") the latter array given the ...
Kevin Kelly's user avatar
3 votes
1 answer
3k views

Difference between Jordan, Elman and normal RNN

As far as I know for history, the Jordan network was proposed first in 1986 as a form of RNN with this diagram: Actually, this is the solution that makes sense when thinking about sequence data that ...
Osama El-Ghonimy's user avatar
2 votes
2 answers
97 views

What if we input sequence data to feedforward network?

One main advantage of RNN is the ability to take input of variable length like the case of sequences. However, what if we neglected this case and assumed some applications that may accept some fixed ...
Osama El-Ghonimy's user avatar
1 vote
1 answer
171 views

How does "one-to-many" RNNs work?

I recently came across an article about RNNs here. Which describes different types of RNNs like: The first figure makes sense. A regular feedforward network. The second is a big question for me. Is ...
Gergő Horváth's user avatar
1 vote
0 answers
50 views

state transition classification on terminal state

I have data on a unit $i$ which enters an entry state $S_0$. This unit has some covariates $x_i$ I would like to predict the probability the unit will reach the terminal state $S_{pos}$ or $S_{neg}$. ...
Hanan Shteingart's user avatar
1 vote
1 answer
209 views

Clarify recurrent neural networks

I'm in the beginning to learn and understand recurrent neural networks. As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and ...
Gergő Horváth's user avatar
8 votes
3 answers
419 views

feature importance after classification

I have time series data and more or less 200 features for each sample, I used a recurrent neural network for the binary classification task. After the classification I would like to know which ...
Rick0's user avatar
  • 105
0 votes
0 answers
48 views

Why is the accuracy on the test dataset very low when training a neural network on an IMU dataset?

I am trying to train an IMU (Inertial Measurement Unit) dataset. The dataset contain 6 features (3-gyro, 3-accelerometer) and 1 label column. I have build a neural network via Conv1D, LSTM and Dense ...
dasmehdix's user avatar
  • 136
0 votes
1 answer
132 views

How can I picture an unfolded RNN as a normal Feed Forward Network?

I am currently working on a Transformer architecture. Trying to picture an RNN (or Encoder) as a normal Feed Forward network really confused me after looking at the following image in an article: (...
Fishie's user avatar
  • 1
0 votes
1 answer
75 views

LSTM / GRU weights during test time

I am working on a historic time series dataset and using RNN, LSTM, GRU models, and I didn't find an answer if in test time, the h (or h, c) weights should be zeors for each batch? If the weights ...
Yuval Asher's user avatar
0 votes
2 answers
2k views

What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
Vistas's user avatar
  • 3
0 votes
1 answer
78 views

Does LSTM without delayed inputs work as a deep net?

I want to predict a multivariate time series. My time series is $a_1(t),...,a_k(t)$ and I want to predict $a_k(t)$. I use the following keras LSTM: ...
Vahid Shams's user avatar
2 votes
1 answer
47 views

GRU and LSTM does not "take risk" predicting

I tested LSTM and GRU models to predict the exchange rate between currencies. I do not take the raw price but a the delta with the previous day, so the data is stationnary around zero. My problem is ...
alarty's user avatar
  • 21

1 2 3
4
5
15