1
$\begingroup$

I'm in the beginning to learn and understand recurrent neural networks. As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time. Not sure if it's a right way to think about it, but so far it's my first impression.

Some things are unclear though.

  1. As far as I understood the final output of each timestep is supposed to predict the input of the next timestep. Is this true? What if I would just like to show the network two images of for example a horse, and depending on them, predict what distance did it go, and in which direction? Is this possible?

From https://towardsdatascience.com/the-most-intuitive-and-easiest-guide-for-recurrent-neural-network-873c29da73c7

  1. In the illustration above there's $A_0$. From where? I would assume at least two timesteps are needed to make a prediction, so in my understanding an $x_0$ is missing from the left side of the diagram. Am I right?

  2. I've been reading through an article which says "Lets train a 2-layer LSTM with 512 hidden nodes". Does it mean two layer of activations, and 512 timesteps?

$\endgroup$
1
$\begingroup$

As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time.

Not really. Each cyan box in your image represents the exact same cell. Now this cell can be a lot of things just take a look at LSTM cell (h and c represents your A) but it can also be a network which takes $A_i$ and $X_{i+1}$ as input and returns $A_{i+1}$ and $Y_{i+1}$ as output.

  1. It may be true if the RNN tries to predict e.g. a time series. To train such a net you would provide the series as a training input and the same time series but in the next step as an output (so it would try to predict $X_{i+1}$ based on $\forall_{j \in [1;i)} X_{j}$. But in general it's not true. The output may be in a completely different format and represent completely different thing exactly like in your example. In your example your $X_i$ is an encoded i-th frame and $Y_i$ is what the network thinks the distance the horse has traveled is up until i-th frame.
  2. $A_0$ is the starting state of the RNN. What it is exactly depends on the exact architecture used but it's common to just set it all to zeros. We need this starting state because as I mentioned the same cell is used at each recurrent step so there has to be something to provide as network state in the beginning. There is no $X_0$ missing. Also there is nothing stopping you from making a prediction based on a sequence of length 1. It's just that it's not useful to use a RNN in such a situation.
  3. The number of timesteps taken depend on the data - not the network. You can use the same network on sequence of length 512, 2 and 2 million. That's why they're commonly used to solve problems with varying length like speech recognition. LSTMs have weights just like a normal neural network does. You can think of these 512 hidden nodes as the size of a hidden layer in the cell. Using two layers of LSTM means using two LSTMs with 512 nodes, and using output of the first as the input of the second one. The output of the second LSTM is the output of 2-layer LSTM.
| improve this answer | |
$\endgroup$
  • $\begingroup$ As you see in the image, there's $w_{aa}$, $w_{ax}$ as the weights. The reason behind they aren't marked as $w_{aa}^1$ (in case of the first timestep) is because - according to your answer as each cyan box is the same cell - the weights are shared? $\endgroup$ – Gergő Horváth Sep 22 at 16:25
  • 1
    $\begingroup$ Correct. It also means that its derivative is calculated multiple times ("length of the sequence" times to be exact) and then summed to compute final weight update. $\endgroup$ – YuseqYaseq Sep 22 at 19:11
  • $\begingroup$ I've met with a RNN example, where to convert 5 timesteps to 3 at the end was solved with a RNN layer, get the last timesteps output, cloned it 3 times, and forwarded to another recurrent layer (then the results forwarded to a dense layer, but the question is not for that). Now knowing that the weights are shared, what was the point in putting a RNN layer to the always exactly same 3 inputs? Why not just directly connect the first RNN layer to a dense one? It actually works worse, but why? $\endgroup$ – Gergő Horváth Sep 22 at 20:16
  • 1
    $\begingroup$ It's hard for me to tell not knowing what this RNN was supposed to solve but it's common to take only the last input from the RNN and ignore the rest. If we return to your horse example - you can see that we don't care about any intermediate $Y$, only the last one.Also it's common to stack multiple RNNs to solve more complex problems. $\endgroup$ – YuseqYaseq Sep 23 at 7:15
  • 1
    $\begingroup$ For example if you're familiar with how compilers are built - they basically move through code a few times, each time turning the code into more and more interpretable representation. It turns out that if you stack RNNs that many times and tell it to turn one code to another these RNNs will simulate the steps of compilation. An effect a dense layer would simply not produce. $\endgroup$ – YuseqYaseq Sep 23 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.