0
$\begingroup$

More often than not, I see RNNs being used with fixed length timesteps. So what is the difference between the following two networks?

  1. RNN with timestep length of 3 over sequence Xt.
  2. NN with inputs x(t-2), x(t-1), x(t)

They both get 3 timesteps of sequence at each turn. Thus, (I know I am wrong but) these two networks have the same capability. They both use previous 3 samples to predict the next sample. What is the difference, then?

$\endgroup$
1
$\begingroup$

It's true that they both use 3 samples to predict the next sample. But the difference is obviously how they learn it. Deep Neural Nets will treat time-distributed samples such as they were single sample at once. However, RNN will learn it as a sequence by its sequential nature, part by part.

Each activation layer of (uni-directional) RNN will take a part of a sequence, make prediction and feed the prediction on that part forward to the next activation layer which will take the next part of the sequence as input. Important part is that the prediction is feeded forward so that the next neuron learning the next part of the sequence will also affected from the prediction of the previous neuron as well as the input sequence, unlike classical neural networks. It can get easy to understand when you look at the visual example below:

enter image description here

Hope I could help.

$\endgroup$
0
$\begingroup$

For intuition, RNN is having more assumption and adding more constraint than MLP and therefore supposed to be more efficient but also more limited. MLP can do same tasks as RNN. You can ead this post

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.