Cross-Validation: Repeated K-Fold/Group K-Fold

Question

Repeated K-Fold vs Group K-Fold

As per my understanding from sklearn docs

Repeated K-Fold:

RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing different splits in each repetition.

Repeated Stratified K-Fold cross validator:

Repeats Stratified K-Fold n times with different randomization in each repetition.

Group K-Fold:

GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.

• Can somebody explain in-detail, When would one use Repeated K-Fold over Group k-fold?
• What are the advantages/disadvantages of using Repeated K-Fold over Group k-fold?

Answer 1

Group k-fold is sufficiently specialized that the comparisons you ask for don't really make sense. "Repeated" really does just mean to remake the splits multiple times; you could easily make a "repeated group k-fold" splitter.

You use group k-fold when you have groups you don't want split across the training and test sets. For example, if your data includes multiple rows for each customer (but it still makes sense to train on individual transactions/rows), and your production use-case involves making predictions for new customers, then testing on rows from customers that also have rows in your training set may be optimistically biased.

You use repeated XYZ when the data and models have high variability and you want more performance datapoints with which to make statistically sound statements. (Likely this is when your dataset is small, in which case happily your model building should be faster so that you can afford the computational cost of repeating the cross-validation procedure multiple times.)

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html
esp. the image:

(but now I'd like to see one with groups not lining up with classes like that...)