# Why don't we gradually update the activation parameters in RNN from one activation to the next as the network is learning more?

I'm very new to (unidirectional, vanilla) RNN and sequence modeling in general, and all I understood about the motivation on having the connection between two successive hidden layers/activation is that: this connection is needed to reuse the information learnt from the $$t$$-th part of the $$i$$-th sequence, $$x_i^{}$$ to learn about the $$x_i^{}$$, i.e. the $$(t+1)$$ -st part of the (same) $$i$$-th sequence.

Correct me if I'm wrong, but I fail to see the motivation behind using the same activation-to-activation parameter set $$\theta_{aa}$$ for every connection between two successive hidden states, except of course that: we have less number of parameters to estimate while we minimize the cost. I can't help thinking that $$\theta_{aa}$$ should be gradually updated with each hidden state, as more information (=part of the same sequence, i.e. words in case of translation) comes in. See the example below.

Let's consider the example of machine translation from English to French for example: (EN) "I am a man" to (FR) "Je suis un homme". Here intuitively, the RNN should try to learn that "am" occurs with certain probability after "I" in English, and correspondingly in French, "suis" occurs with certain probability after "Je"; but given that it's already learnt that, the (conditional?) probability of the occurrence of "un homme" after "je suis" can be more effectively estimated when we know the probability of "a man" occurring after "I am". So intuitively, the RNN should be "better informed" when it knows more parts of the given sequence than lesser part of it, and hence the activation parameters should be gradually updated accordingly.

I must be missing something, but not sure what it is? I've only motivated myself using the machine translation example, but examples from other areas would also be appreciated.

The motivation to use RNN is that the length of the sequence or position info is random in the data.

For example we could use the sample trained RNN model to translate the following setences:

1. I am a man
2. I am a woman and you are a man

In RNN, we do not consider the position of the words, only consider the relation between words. Thus the different activation parameters trained for different positions are useless.

Moreover to make RNN better informed (using previous words / next words) we could use Gated recurrent unit (or LSTM) and Bidirectional RNN.

A RNN is supposed to be able to accumulate information across the sequence, as you suggest. Each time it observes a new token, it combines that token with its previous hidden state. It then incorporates information about that token into the hidden state and produces a new hidden state. The idea is that the hidden state summarizes information about all tokens seen so far.

The reality, however, is a bias toward recent tokens. LSTMs ameliorate that problem by modeling how much information in the hidden state to keep at each step. Nonetheless they will tend to lose information over long stretches.

Note that many applications of RNNs don't just use RNNs, but use RNNs followed by an attention mechanism. The attention sees all hidden states that the RNN produces. It can then "look back" at any hidden state, allowing information about the sequence to be used even if it isn't retained in the RNN's final hidden state.