4
$\begingroup$

Repeated K-Fold vs Group K-Fold

As per my understanding from sklearn docs

Repeated K-Fold:

RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing different splits in each repetition.

Repeated Stratified K-Fold cross validator:

Repeats Stratified K-Fold n times with different randomization in each repetition.

Group K-Fold:

GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.

  • Can somebody explain in-detail, When would one use Repeated K-Fold over Group k-fold?
  • What are the advantages/disadvantages of using Repeated K-Fold over Group k-fold?
$\endgroup$

1 Answer 1

4
$\begingroup$

Group k-fold is sufficiently specialized that the comparisons you ask for don't really make sense. "Repeated" really does just mean to remake the splits multiple times; you could easily make a "repeated group k-fold" splitter.

You use group k-fold when you have groups you don't want split across the training and test sets. For example, if your data includes multiple rows for each customer (but it still makes sense to train on individual transactions/rows), and your production use-case involves making predictions for new customers, then testing on rows from customers that also have rows in your training set may be optimistically biased.

You use repeated XYZ when the data and models have high variability and you want more performance datapoints with which to make statistically sound statements. (Likely this is when your dataset is small, in which case happily your model building should be faster so that you can afford the computational cost of repeating the cross-validation procedure multiple times.)

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html
esp. the image:
sklearn docs visualization of group k-fold
(but now I'd like to see one with groups not lining up with classes like that...)

$\endgroup$
2
  • $\begingroup$ Apologize if it is obvious, but can you elaborate on the second paragraph? Why "testing on rows from customers that also have rows in your training set may be optimistically biased"? $\endgroup$
    – ado sar
    Aug 24, 2023 at 23:51
  • $\begingroup$ @adosar It's clear with an extreme example: some features together uniquely identify customers, and some others determine the target in a way that depends on the customer. Then the model can perform very well on customers it's already seen (identify the customer then compute their target), but may perform very poorly on new customers (it hasn't learned how their targets are determined from the latter features). $\endgroup$
    – Ben Reiniger
    Aug 25, 2023 at 2:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.