Just when I thought I had convinced myself that RNNs make no other assumption about a sequence other than that there are dependencies between the inputs and that (in the case of monodirectional RNNs) the past affects the present, Goodfellow, Bengio and Courville (2016) hit me with this:
"The parameter sharing used in recurrent networks relies on the assumption that the same parameters can be used for different time-steps. Equivalently, the assumption is that the conditional probability distribution over the variables at time t + 1 give the variables at time t is stationary, meaning that the relationship between the previous time step and the next time step does not depend on t."
Could someone elaborate on what this means with regards to the stationarity assumption for the time series?
(I have the feeling that there is a notion of local or "input-conditioned" stationarity here but that is just my intuition.)