Measure performance of classification model for training on different snapshots

Question

I am trying to do binary classification on some chronological data. Let's assume we have weekly data from the first week of 2017 through the last week of 2020. Now we have found out that 26 weeks of training data might be sufficient for doing prediction immediate next week. So if I want to do predictions for the 32nd week of 2020, my training window will be from the 6th week of 2020 to the 31st week of 2020.

Now I am taking the last 16 weeks from 2020 and training a model for each of them. Each time 26 preceding weeks constitute the training data. I have three doubts:

1. How to report the overall model accuracy in this case? I am taking the average model performance over 16 snapshots, but the performances are really different in different snapshots (some of them have AUC 76, Some have 58).
2. I am keeping the same set of hyperparameters across different snapshots. So technically, can I say that I am using the same model?
3. Let's say my production environment is not having a weekly model retraining facility. Now how to handle this situation? How do choose a model to do predictions for upcoming snapshots?

Answer 1

1. The average performance is a good summary of the performance, but you should also mention the variations across time. For example you could calculate the standard deviation across the snapshots.
2. I think it would be confusing to say that it's the same model, normally the same model would mean single training. Instead you should explain that the exact same method (including same hyper-parameters) is used to train every snapshot model.
3. Red flag here: if the production environment doesn't make it possible to use the same ML design, then this design is pointless. Either the production environment can be changed, or you need a completely different strategy.