I have seen pictures of RNNs and LTSMs, and they usually look like this: enter image description here Here the task is to take a sentence and make a prediction of some sort. What are each of the green squares? Are each of them layers, or does each green square have it's own set of layers and neurons?

Additionally, in such a scenario, the input data is the entire sequence of words. When dealing with time series data, would the input be the entire data set since the entire dataset is a sequence? If you have 1million observations, you would need to construct 1million green squares?

  • $\begingroup$ Every green square is a iteration/forward-pass on the model. It uses the same model over and over again feeding it the input data and the previously output data. $\endgroup$ – Pedro Henrique Monforte Aug 4 at 2:39

The architecture of a RNN is called recurrent because it applies the same function at each step. So all the cells on the graph actually represent the same computation, but not the same state. Each green square in your figure represent the computation. $$ s^{(t)} = f(s^{(t-1)}, x^{(t)}, \theta) $$ Where $f$ is the function of the RNN, $\theta$ are parameters, $s^{(t)}$ is the RNN state at step $t$ and $x^{(t)}$ is the input at step $t$ in the input sequence.

What you see represented in the figure is actually what is called the unfolded representation, that is the same RNN cell applied to the input sequence one input vector at a time.

I recommend you to read the chapter 10 on RNN Deep Learning book. The following figure is from this book and summarize the idea.

enter image description here

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.