How to use sklearn train_test_split to stratify data for multi-label classification?

Question

I am attempting to mirror a machine learning program by Ahmed Besbes, but scaled up for multi-label classification. It seems that any attempt to stratify the data returns the following error: The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2.

In my data set, I have 1 column which contains clean, tokenized text. The other 8 columns are for the classifications based on the content of that text. Just to note, column 1 - 4 have significantly more samples than 5 - 8 (more obscure classifications derived from the text).

Here is a generic sample from my code:

x = data['cleaned_text']
y = data[['car','truck','ford','chevy','black','white','parked', 'driving']]

x_train, x_test, y_train, y_test = train_test_split(x,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=42)

print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)

Output: (6293,) (700,) (6293, 8) (700, 8)

Adding stratify=y to train_test_split returns the error previously mentioned. Even when I limit y to just one column, I still get the error.

How can I stratify the data so that I give the program a fair look in the training set?

Answer 1

Try this:

from skmultilearn.model_selection import iterative_train_test_split X_train, y_train, X_test, y_test = iterative_train_test_split(x, y, test_size = 0.1)

Since you're doing multilabel classification, it's very likely to get unique combinations of each class, which is what causes the error with sklearn. You have to use a special library for multilabel stratified splitting.

More details on how to use skmultilearn

Answer 2

The error you're getting indicates it cannot do a stratified split because one of your classes has only one sample. You need at least two samples of each class in order to put one in the training split and one in the test split. You should examine what your class breakdown is to find the culprit.

Answer 3

There is a seperate module for classes stratification and no one is going to suggest you to use the train_test_split for this. This could be achieved as follows:

from sklearn.model_selection import StratifiedKFold


train_all = []
evaluate_all = []
skf = StratifiedKFold(n_splits=cv_total, random_state=1234, shuffle=True)
for train_index, evaluate_index in skf.split(train_df.index.values, train_df.coverage_class):
    train_all.append(train_index)
    evaluate_all.append(evaluate_index)
    print(train_index.shape,evaluate_index.shape) # the shape is slightly different in different cv, it's OK

# Getting each batch
def get_cv_data(cv_index):
    train_index = train_all[cv_index-1]
    evaluate_index = evaluate_all[cv_index-1]
    x_train = np.array(train_df.images[train_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    y_train = np.array(train_df.masks[train_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    x_valid = np.array(train_df.images[evaluate_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    y_valid = np.array(train_df.masks[evaluate_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    return x_train,y_train,x_valid,y_valid

# Training loop
for cv_index in range(cv_total):
    x_train, y_train, x_valid, y_valid =  get_cv_data(cv_index+1)
    history = model.fit(x_train, y_train,
                        validation_data=[x_valid, y_valid], 
                        epochs=epochs)

This is a simple code snippet for using StratifiedKFold with your code. Just replace the required parameters and hyper-parameters accordingly.

Answer 4

As said by @chenjesu: what you really want is likely to consider each labels when performing stratification (rather than only the combinations of labels which are often only seen once).

Unfortunately, the scikit_multilearn function that was pointed out is extremely slow for medium to large-sized datasets. E.g. 3 min for 100k examples (and never seems to finish if you use n_labels=20):

# pip install scikit-multilearn
from sklearn.datasets import make_multilabel_classification
X,Y = make_multilabel_classification(n_samples=100000, n_classes=100, n_labels=10)

# %%time
from skmultilearn.model_selection import iterative_train_test_split
X_train, y_train, X_test, y_test = iterative_train_test_split(X,Y,test_size=0.20)
# CPU times: total: 2min 45s

I would use instead use the following function that uses the iterative-stratification package. This only requires 2 seconds on the same data:

# pip install iterative-stratification
from sklearn.datasets import make_multilabel_classification
X,Y = make_multilabel_classification(n_samples=100000, n_classes=100, n_labels=10)

%%time
X_train, y_train, X_test, y_test = multilabel_train_test_split(X,Y,stratify=Y, test_size=0.20)
# CPU times: user 2.31 s

where:

from iterstrat.ml_stratifiers import MultilabelStratifiedShuffleSplit
from sklearn.utils import indexable, _safe_indexing
from sklearn.utils.validation import _num_samples
from sklearn.model_selection._split import _validate_shuffle_split
from itertools import chain

def multilabel_train_test_split(*arrays,
                                test_size=None,
                                train_size=None,
                                random_state=None,
                                shuffle=True,
                                stratify=None):
    """
    Train test split for multilabel classification. Uses the algorithm from: 
    'Sechidis K., Tsoumakas G., Vlahavas I. (2011) On the Stratification of Multi-Label Data'.
    """
    if stratify is None:
        return train_test_split(*arrays, test_size=test_size,train_size=train_size,
                                random_state=random_state, stratify=None, shuffle=shuffle)
    
    assert shuffle, "Stratified train/test split is not implemented for shuffle=False"
    
    n_arrays = len(arrays)
    arrays = indexable(*arrays)
    n_samples = _num_samples(arrays[0])
    n_train, n_test = _validate_shuffle_split(
        n_samples, test_size, train_size, default_test_size=0.25
    )
    cv = MultilabelStratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=123)
    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )

Answer 5

If you are using pandas, you could try this:

• Explode on y -> e.g. df.explode('labels')
• Performe regular train_test_split with stratification
• Regroup on 'cleaned_text' & aggregate over labels -> e.g. df.groupby('cleaned_text').agg({'labels': 'sum')

Answer 6

This is because one of your class has only one record. train_test_split is unable to decide where to put that in train or test part. you can do either of following:

1. you create one/more copy of that record or
2. remove from the main data or
3. keep that in a variable and then remove it, then do train_test_split and append it to training data