0
$\begingroup$

I'm very new to (unidirectional, vanilla) RNN and sequence modeling in general, and all I understood about the motivation on having the connection between two successive hidden layers/activation is that: this connection is needed to reuse the information learnt from the $t$-th part of the $i$-th sequence, $x_i^{<t>}$ to learn about the $x_i^{<t+1>}$, i.e. the $(t+1)$ -st part of the (same) $i$-th sequence.

Correct me if I'm wrong, but I fail to see the motivation behind using the same activation-to-activation parameter set $\theta_{aa}$ for every connection between two successive hidden states, except of course that: we have less number of parameters to estimate while we minimize the cost. I can't help thinking that $\theta_{aa}$ should be gradually updated with each hidden state, as more information (=part of the same sequence, i.e. words in case of translation) comes in. See the example below.

Let's consider the example of machine translation from English to French for example: (EN) "I am a man" to (FR) "Je suis un homme". Here intuitively, the RNN should try to learn that "am" occurs with certain probability after "I" in English, and correspondingly in French, "suis" occurs with certain probability after "Je"; but given that it's already learnt that, the (conditional?) probability of the occurrence of "un homme" after "je suis" can be more effectively estimated when we know the probability of "a man" occurring after "I am". So intuitively, the RNN should be "better informed" when it knows more parts of the given sequence than lesser part of it, and hence the activation parameters should be gradually updated accordingly.

I must be missing something, but not sure what it is? I've only motivated myself using the machine translation example, but examples from other areas would also be appreciated.

$\endgroup$
0
$\begingroup$

The motivation to use RNN is that the length of the sequence or position info is random in the data.

For example we could use the sample trained RNN model to translate the following setences:

  1. I am a man
  2. I am a woman and you are a man

In RNN, we do not consider the position of the words, only consider the relation between words. Thus the different activation parameters trained for different positions are useless.

Moreover to make RNN better informed (using previous words / next words) we could use Gated recurrent unit (or LSTM) and Bidirectional RNN.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.