Do Recurrent Neural Networks assume stationarity or just a general kind of sequential dependence?

Question

Just when I thought I had convinced myself that RNNs make no other assumption about a sequence other than that there are dependencies between the inputs and that (in the case of monodirectional RNNs) the past affects the present, Goodfellow, Bengio and Courville (2016) hit me with this:

"The parameter sharing used in recurrent networks relies on the assumption that the same parameters can be used for different time-steps. Equivalently, the assumption is that the conditional probability distribution over the variables at time t + 1 give the variables at time t is stationary, meaning that the relationship between the previous time step and the next time step does not depend on t."

Could someone elaborate on what this means with regards to the stationarity assumption for the time series?

(I have the feeling that there is a notion of local or "input-conditioned" stationarity here but that is just my intuition.)

Thanks!

Answer 1

"The parameter sharing used in recurrent networks relies on the assumption that the same parameters can be used for different time-steps

For each time-step, the hidden state i.e., "memory" of the RNN is calculated as

$$ h_t = f(U * x_t + W * h_{t−1})$$ For $n$ tokens in our input sequence that would look like:

## Pseudocode for simple rnn

initialise(hidden, U, W, V)
# forward pass
for t in range(sequence_length):
    hidden[n] = rnn_forward(x[t], W, U, hidden[n-1])

# backpropagation tt
for t in range(sequence_length):
    dW_t, dU_t = rnn_backward(hidden[-t], U, W)
    dW += dW_t 
    dU += dU_t 
optimizer.step(dW, dU)
    

What you can see in the process above is that the RNN layer shares the same parameters $(U, V, W)$ across all steps. It may help to think the RNN in its unrolled version (figure below). This reflects the fact that we are performing the same task at each step inside a for loop, just with different inputs. This greatly reduces the total number of parameters we need to learn.

Anyway, hope this helps.

References:

• https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-1/
• https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

Answer 2

The distribution is stationary, but it doesn't mean the time series should be stationary. Let's imagine the recurrent network, that returns the n+1 if it was n on previous step. Distribution is not changing with time, but the time-series always goes up.