2
$\begingroup$

All the descriptions of RNNs introduce some equation like:

$\ h_t = f_W(h_{t-1},x_t)$

and I'm wondering why we don't just go straight to the "source", ie the last input like :

$\ h_t = f_W(x_{t-1},x_t)$

The only thing I can maybe think of is maybe the RNN can store some sort of "state" in ht-1

But then, along the same vein, what would happen in an LSTM architecture if we used previous inputs vs previous outputs? It seems to me that the presence of the memory functionality should be able to avoid this potential problem (if it even is one)

$\endgroup$
3
$\begingroup$

That would work if the Markov Assumption was in place here (i.e. the last state was enough to determine the new state).

However, that is exactly what RNNs try to break. RNNs try to encode the whole history of inputs and predictions to determine the next output.

So in the formula, the $h_{t-1}$ is the last output, but it is a hidden state which encodes all the previous inputs, outputs, and calculations. So when you have new information coming in (i.e. $x_t$), your model can make use of all the history and the new input to determine the new input.

The same applies to LSTM, you are working with a sequence that the new output depends not only on the last input but on more things from the past, and this information is encoded in $h_{t-1}$.

$\endgroup$
  • $\begingroup$ That totally makes sense. Thanks a lot! Also one more thing to add that I just realized from watching this video youtube.com/watch?v=6niqTuYFZLQ , the whole premise of my questions kinda falls apart when there's only one input and the network is trying to generate multiple outputs. $\endgroup$ – 1mike12 Aug 12 '18 at 18:34
  • $\begingroup$ @1mike12 that also makes a lot of sense! You cannot input what you do not have ;) $\endgroup$ – BrunoGL Aug 12 '18 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.