I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for the model (n days), and the model parameters.
For each day, we train model on the past N days, and predict value of Y for next day. We keep rolling through the data, and record forecasting metric on the predicted day (Absolute Error)
Thus, for each combination of parameters P, we have an average Absolute Error across the predicted days, and by trying many different combinations of parameters P, we can find which parameters P minimize prediction error.
This is the current methodology I am aware of, and am usually using:
1) We split the data in time into 3000 days(Train-test) + 1000 days(Validation)
2) On the (Train-test) dataset, we find parameters P that minimize test forecasting error. Train is the last N days of data, Test is the next day, and we keep rolling the 'Test' window over the first 3000 days of data. 3) We now have optimal parameters P from the (Train-test) set computations. We now use those parameters P on the 1000 points Validation set, and record final performance metric on that Validation set.
My 2 questions are:
1) Aren't we biased by the data split? Suppose we had split into 2000 (Train-test) and 2000 (Validation) and we found different optimal parameters P over the first (Train-test) set. How do I decide which parameters are optimal for production? The parameters from the 2000/2000 or the 3000/1000 split? I followed (Train-test) and (Validation) in both cases, and yet I find different optimal parameters, just because I spit data differently.
2) Are there any papers on how to concretely apply time-series forecasting in practice? Most papers use a simple Train-Test logic, with no validation and parameter optimization so not sure where to look.