Train/validation/test and cross-validation on panel dataset

Question

(Cross-posting a previous question from CrossValidated in case it is more suitable here: Train/Validation/Test and Cross-Validation on Panel Dataset )

I have a panel dataset, indexed by $Year$ and $Country$, along with input features $X_{1},..,X_{N}$ and the binary output target $Y$. Suppose 20 countries and years spanning from 2000 to 2020

Country, Year,  X1, X2, ...,  XN,  Y
USA,     2000, 0.1, 0.5, ..., 0.4, 1
USA,     2001, 0.2, 0.4, ..., 0.5, 0
USA,     2002, 0.3, 0.3, ..., 0.4, 0
China,   2000, 0.4, 0.2, ..., 0.3, 1
China,   2001, 0.5, 0.1, ..., 0.3, 1
China,   2002, 0.6, 0.3, ..., 0.4, 1
...

My purpose is to prepare a predictive model $f(X_1, ..,X_N) = \hat{Y}$, while at the same time following a scientifically correct method of model selection and evaluation. Supposing the data were standard iid data, as far as I know there would be two standard approaches:

• Train/Validation/Test: Fit all model specifications on Train, select the best model specification $M$ on Validation (i.e. the best performing hyperparameters), retrain the model on (Train+Validation) and perform the final evaluation on Test
• Cross-Validation and Test: Perform cross-validation on a folded Train to get the best model specification $M$, retrain the model on Train (all folds) and perform final evaluation on Test.

Now my questions are centered around what would be the analogous procedures for panel data. Initially, I 've followed this approach:

Method 1: Starting from 2000, perform standard cross-validation on $k$ years of data and test on the next year. Repeat the procedure for the next year until the final year. For k=5, that would be:

Method 1 (k=5):

Standard CV     Test
2000-2004       2005
2001-2005       2006
2002-2006       2007
...
2015-2019       2020

In the above example, the final evaluation would be defined as the average performance on test years 2005-2020. A possible problem is that the best model specification $M$ selected from the 2000-2004 CV may differ from the best specification $M'$ of 2001-2005 CV, and so on. So, different hyper-parametrizations of the same algorithm will be used for the scoring of each test set. Is this a bad practice, if my purpose is to compare the performance of different algorithms?

Another question, is it acceptable to use Random CV to derive the model specification as long as my out-of-sample test sets come after the data used for cross-validation? For instance, while in the first iteration the 2000-2004 data will get mixed up in random folding, these data still precede the independent test data of 2005. If this is indeed problematic, a different approach would be to replace random CV with time-series CV (see here), to get a modified version of the method:

Method 1.2 (k=5):

Timeseries CV   Test
2000-2004       2005
2001-2005       2006
2002-2006       2007
...
2015-2019       2020

While this revision solves the second problem (mixing up time in CV), there is still the issue of multiple model parametrizations. To explain, while I may say that e.g. SVM had an average accuracy of 0.9 across the tests 2005, ... , 2020, the SVM hyperparameters may not be the same for all test sets, as they are specified from different CV executions. If, lets say, Random Forest had 0.8 average accuracy, I wonder if it is right to assume that SVM is better for this problem, while at the same time I can not provide a specific set of hyperparameters for either SVM or Random Forest.

The only way I can think of to alleviate the multiple parametrizations problem would be something along the lines of Method 2. The idea is to keep the last $m$ years for testing and the preceding years for time series CV. The multiple parameters problem is solved because CV is only performed once, so there is only one specification. However, there is only one test set, and I find it pessimistic in the sense that 2017-2018 data are not used for hyperparameter tuning when we test on 2019 and 2020 etc. (Well, the model could be at least retrained including 2017-2018 without changing the hyperparameters specified by 2000-2016 CV)

Method 2 (m=4)

Timeseries CV   Test
2000-2016       2017-2020

Finally, regarding the $k$ and $m$ parameters of the aforementioned methods, would it be cheating to try different values in order to get the best test score? e.g. For Method 1, to experiment with k=2,3,4,5..

to wrap it up

• is it ok to shuffle time data as long as they are used for hyperparameter tuning cv but not testing, or is time-series CV necessary?
• Are there more standard/scientifically acceptable methods than the three I described?
• is the "multiple specifications" issue a real problem, if I only want to compare different learning algorithms without providing hyperparameter specification?
• I have not used the $Country$ variable at all in any of my methods. Would it be a good idea, or even necessary, to do so?
• While I will greatly appreciate direct answers, feel free to share any resources, papers or books on the correct approach to do train/validation/test with cross-validation on panel data