I'm looking for references on training time-series regression RNN models. For learning purposes I want to implement myself using autograd (or JAX) rather than a high level library. I cannot find really good references on how to train this class of model.

Specifically I have 1 years data at 15 minute resolution. The model I'm fitting is

x[t+1] = f(x[t],u[t])
y[t] = g(x[t])

where x is the latent state, u the input and y the output at time t.

Assuming no gaps in the data, I need to learn the parameters of f and g, and optionally the initial state (or provide a guess). One way of doing this is to compute derivatives and the loss on the entire sequence from t0 to T. In this case I can either initialise x[0]=0 and hope it's influence decays to zero or learn it.

Another way is to batch into say daily subsequences. This seems to be the more common method (since it aligns with stochastic gradient descent), but how then do you deal with the initial state for each subsequence? One way is to set x[0]=0, update the derivatives using the first subsequence, predict the subsequence and roll the final state from subsquence i to the initial state for subsequences i+1? Another is to attempt to learn the average initial state for each subsequence - it feels like this might work but only if the subsequences are long enough that the error in the intitial state estimation decays relatively quickly relative to the length of the subsequence.

I've not really found any really good papers or even blogs on this though, most blogs are aimed at showing how to implement an RNN using a high level api and don't cover the low level details or are talking about independent sequences such as sentences rather than a time-series.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.