Repeated k-fold Cross Validation for time series data?

Question

I have a relative small sample size (330 with 45 features) + it's time series data.

I want to train my LightGBM regression model for best generalized RMSE score and want to use repeated CV. I use hyperopt to do hyperparameter optimization to optimize for lowest RMSE.

The last 2 months include data post-corona, so a standard k-fold CV would probably fail when testing on such data because there was quite a shift in the target variable y.

Standard ways to do repeated CV use resample / reshuffle which is not useable with time series data.

What is best practice in this case? How can I do repeated Cross Validation while working with time series data?

At the moment I do this for the fmin() function.

def lightgbm_cv_repeated(params):
    params = {
        'n_estimators': int(params['n_estimators']), 
        'max_depth': int(params['max_depth']), 
        'learning_rate': params['learning_rate'],
        'min_child_samples': int(params['min_child_samples']),
        'min_child_weight': params['min_child_weight'],
        'feature_fraction': params['feature_fraction'],
        'bagging_fraction': params['bagging_fraction'],
        'bagging_freq': int(params['bagging_freq']),
        'num_leaves': int(params['num_leaves']),
        'max_bin': int(params['max_bin']),
        'num_iterations': int(params['num_iterations']),
        'objective': 'rmse',
        }

    scores = []

    for i in range(5, 11):
        print(i)
        cvTSS = TimeSeriesSplit(max_train_size=None, n_splits=i)
    
        model = lgb.LGBMRegressor(random_state=i, **params)
        score = -cross_val_score(model, X=X, y=y, cv=cvTSS, scoring="neg_root_mean_squared_error", n_jobs=-1).mean()
        scores.append(score)

    result = mean(scores)
    return result

Answer 1

No, you can't do regular repeated Cross Validation in time series data. For time series data you might be looking for time series split such as in https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html. BTW, if you have a structural break in your data such as covid, or you can add external data in order to account something like this such as train your data in another crisis event such as subprime one. But overall if your test set is really different from your train set isn't likely that any machine learning algorithm will work.