# Clarify recurrent neural networks

I'm in the beginning to learn and understand recurrent neural networks. As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time. Not sure if it's a right way to think about it, but so far it's my first impression.

Some things are unclear though.

1. As far as I understood the final output of each timestep is supposed to predict the input of the next timestep. Is this true? What if I would just like to show the network two images of for example a horse, and depending on them, predict what distance did it go, and in which direction? Is this possible?

1. In the illustration above there's $$A_0$$. From where? I would assume at least two timesteps are needed to make a prediction, so in my understanding an $$x_0$$ is missing from the left side of the diagram. Am I right?

2. I've been reading through an article which says "Lets train a 2-layer LSTM with 512 hidden nodes". Does it mean two layer of activations, and 512 timesteps?

Not really. Each cyan box in your image represents the exact same cell. Now this cell can be a lot of things just take a look at LSTM cell (h and c represents your A) but it can also be a network which takes $$A_i$$ and $$X_{i+1}$$ as input and returns $$A_{i+1}$$ and $$Y_{i+1}$$ as output.
1. It may be true if the RNN tries to predict e.g. a time series. To train such a net you would provide the series as a training input and the same time series but in the next step as an output (so it would try to predict $$X_{i+1}$$ based on $$\forall_{j \in [1;i)} X_{j}$$. But in general it's not true. The output may be in a completely different format and represent completely different thing exactly like in your example. In your example your $$X_i$$ is an encoded i-th frame and $$Y_i$$ is what the network thinks the distance the horse has traveled is up until i-th frame.
2. $$A_0$$ is the starting state of the RNN. What it is exactly depends on the exact architecture used but it's common to just set it all to zeros. We need this starting state because as I mentioned the same cell is used at each recurrent step so there has to be something to provide as network state in the beginning. There is no $$X_0$$ missing. Also there is nothing stopping you from making a prediction based on a sequence of length 1. It's just that it's not useful to use a RNN in such a situation.
• As you see in the image, there's $w_{aa}$, $w_{ax}$ as the weights. The reason behind they aren't marked as $w_{aa}^1$ (in case of the first timestep) is because - according to your answer as each cyan box is the same cell - the weights are shared? – Gergő Horváth Sep 22 at 16:25
• It's hard for me to tell not knowing what this RNN was supposed to solve but it's common to take only the last input from the RNN and ignore the rest. If we return to your horse example - you can see that we don't care about any intermediate $Y$, only the last one.Also it's common to stack multiple RNNs to solve more complex problems. – YuseqYaseq Sep 23 at 7:15