Mulitlabel stratified k-fold splitting with non-overlapping groups

Question

For multilabel stratification, we have a good solution implemented by scikit-multilearn which I believe is based on the algorithm presented in "On the Stratification of Multi-label Data". However, in some cases when there is a lack of independence for some of the samples in our dataset, we need the split data to contain non-overlapping groups. For example, if we have 10 samples from group A and 5 samples from group B, all the group A samples need to be in one of the split datasets and all the samples from group B need to be in the other set. Does anyone know of an algorithm to achieve this or an approximation of this? I can generate training and testing sets with disjoint groups or I can create label stratified training and testing sets, but I haven't found a good way to do both.

Answer 1

One approach to achieve this would be to use a combination of both label stratification and group-based stratification.

First, you can use label stratification to ensure that the distribution of labels is balanced across the training and testing sets. Then, you can use group-based stratification to ensure that the samples from each group are non-overlapping in the training and testing sets.

One way to do this would be to first identify the groups in your dataset, and then divide the samples in each group into two sets: one for training and one for testing. Then, you can combine the training sets from each group to form your final training set, and do the same for the testing sets.

Another approach could be to use a modified version of K-fold cross-validation where you can make sure that the samples from a certain group are always in the same fold.

It is important to note that there is no one perfect solution for this problem, it depends on the specific characteristics of the dataset and the goal of the classification task.