How to test dev set on Time Series data via forecasting

Question

I'm implementing $3$ Bayesian Deep Learning models (links below) for my masters. I'm supposed to test them on a civil engineering time series data.

My models should take a time series covariate vector ($X_t$ = {$x1_t$, $x2_t$, ...}) as input and predict single values $y_t$. I will use past values of $X$, (e.g. $X_{t-1}$, $X_{t-2}$) on each $y_t$ but the models won't be feed with past values of $y_t$ because these will not be available on a real situation. The data was generated daily, so every t is a new day. Summing up, the model shall take windows of X values (e.g. $X_{t-1}$, $X_{t-2}$ ...) with a fixed window size, and them predicting a single y value. We will have multiple pairs $((X_{t},X_{t-1}),y_t)$ as our training data, for multiple values of $t$.

I have several years of data, and I've separated the data on some arbitrary point T so that the first years are used as training, and the last months of the last year of data are used as test/dev data.

So, training examples are generated until $T (X_T,y_T)$, and them $T+1, T+2...$ are used to test the models.

The only value that matters on this forecasting task is the next day value ($y_{T+1}$).

Initially, I was training the model until day T, and then using that model to predict the annotations $y_{T+n}$ until the dev data stops. But then the people that gave me the data asked about retraining the model every new day of the dev set, and then predicting the next day:

For example, on the first training, I would train until day T and then forecast T+1. Then I would add the real value of $y_{T+1}$ to the train set, retrain the models, and forecast T+2 and so on.

Which approach would be more proper? The accuracy of the models drop when I try to retrain the models every day to predict just a single day, I can't understand why.

I'm using DeepAR, DeepFactors, Enc-Dec-Forecaster (by Uber) and an ARIMA model just to compare. I'd really appreciate some help!

EDIT: For clarity

Answer 1

I feel your frustration. I am not sure if this is a complete answer, but I’ll try to add some value here.

Disclaimer: I am not familiar with the neural nets you mentioned but have used other times series nns. So my answers will be general.

First, there is nothing specifically wrong about retraining every day unless you are doing so at the expense of the number of training epochs. For example, you could conceivably need to train for 20k epochs to get the best predictive power. If this takes 20 hours, it may not feasible. So you may be tempted to train for 5000 epochs. This is probably a reason not to retrain every day. My recommendation would be to invest in more epochs and retrain less frequently. Hopefully the resulting model is more generalized and powerful in the long run.

If you DO still want to train every day, I would load the weights and biases from the previous training session and continue training for a smaller number number of epochs. Start with as many epochs as you can to produce the first model. Then add the additional row and continue training for a manageable number of epochs. TBH, one extra row should not really change your model weights and biases significantly unless the new y is pretty extreme. In fact, I doubt it would even be obvious.

Now, why does your accuracy drop with the new data? Several ideas:

1. Just stating the obvious. In case it’s not. You will need many training epochs. Each application is different, but sometimes people only train for a few epochs and don’t understand why the model sucks. You might need 200 epochs, you might need 2000000.
2. You are overfitting and the addition of the new data is generalizing the model a little more each day. Doubtful, but possible.
3. You are reinitializing your model with random weights and biases and do not have a seed set. So your model could be finding different local minimums each time you train. I would be surprised if it was always in the negative, but it may be that you have just flipped heads 25 times in a row. I would set a seed when initialize to take this out of the picture.

Hopefully there’s something of use in here.