I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of training data might be sufficient for doing prediction immediate next week. So if I want to do predictions for the 32nd week of 2020, my training window will be from the 6th week of 2020 to the 31st week of 2020.
Now I am taking the last 16 weeks from 2020 and training a model for each of them. Each time 26 preceding weeks constitute the training data. I have three doubts:
- How to report the overall model accuracy in this case? I am taking the average model performance over 16 snapshots, but the performances are really different in different snapshots (some of them have AUC 76, Some have 58).
- I am keeping the same set of hyperparameters across different snapshots. So technically, can I say that I am using the same model?
- Let's say my production environment is not having a weekly model retraining facility. Now how to handle this situation? How do choose a model to do predictions for upcoming snapshots?