13
$\begingroup$

I am attempting to mirror a machine learning program by Ahmed Besbes, but scaled up for multi-label classification. It seems that any attempt to stratify the data returns the following error: The least populated class in y has only 1 member, which is too few. The minimum number of labels for any class cannot be less than 2.

In my data set, I have 1 column which contains clean, tokenized text. The other 8 columns are for the classifications based on the content of that text. Just to note, column 1 - 4 have significantly more samples than 5 - 8 (more obscure classifications derived from the text).

Generic sample of the data I am working with

Here is a generic sample from my code:

x = data['cleaned_text']
y = data[['car','truck','ford','chevy','black','white','parked', 'driving']]

x_train, x_test, y_train, y_test = train_test_split(x,
                                                    y,
                                                    test_size=0.1,
                                                    random_state=42)

print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)

Output: (6293,) (700,) (6293, 8) (700, 8)

Adding stratify=y to train_test_split returns the error previously mentioned. Even when I limit y to just one column, I still get the error.

How can I stratify the data so that I give the program a fair look in the training set?

$\endgroup$
2
  • $\begingroup$ You try to predict more than one class at the same time. It's not a multi-class classification, but a multi-label classification problem. Please add a sample of your dataset since it is not clear what you try to do. $\endgroup$
    – Tasos
    Commented Feb 6, 2019 at 16:57
  • $\begingroup$ Thanks, I edited the title and body of the initial question to reflect multi-label vice multi-class. As for the data, I can give a generic example $\endgroup$ Commented Feb 6, 2019 at 17:08

6 Answers 6

11
$\begingroup$

Try this:

from skmultilearn.model_selection import iterative_train_test_split

X_train, y_train, X_test, y_test = iterative_train_test_split(x, y, test_size = 0.1)

Since you're doing multilabel classification, it's very likely to get unique combinations of each class, which is what causes the error with . You have to use a special library for multilabel stratified splitting.

More details on how to use skmultilearn package

$\endgroup$
5
$\begingroup$

The error you're getting indicates it cannot do a stratified split because one of your classes has only one sample. You need at least two samples of each class in order to put one in the training split and one in the test split. You should examine what your class breakdown is to find the culprit.

$\endgroup$
3
$\begingroup$

As said by @chenjesu: what you really want is likely to consider each labels when performing stratification (rather than only the combinations of labels which are often only seen once).

Unfortunately, the scikit_multilearn function that was pointed out is extremely slow for medium to large-sized datasets. E.g. 3 min for 100k examples (and never seems to finish if you use n_labels=20):

# pip install scikit-multilearn
from sklearn.datasets import make_multilabel_classification
X,Y = make_multilabel_classification(n_samples=100000, n_classes=100, n_labels=10)

# %%time
from skmultilearn.model_selection import iterative_train_test_split
X_train, y_train, X_test, y_test = iterative_train_test_split(X,Y,test_size=0.20)
# CPU times: total: 2min 45s

I would use instead use the following function that uses the iterative-stratification package. This only requires 2 seconds on the same data:

# pip install iterative-stratification
from sklearn.datasets import make_multilabel_classification
X,Y = make_multilabel_classification(n_samples=100000, n_classes=100, n_labels=10)

%%time
X_train, y_train, X_test, y_test = multilabel_train_test_split(X,Y,stratify=Y, test_size=0.20)
# CPU times: user 2.31 s

where:


from iterstrat.ml_stratifiers import MultilabelStratifiedShuffleSplit
from sklearn.utils import indexable, _safe_indexing
from sklearn.utils.validation import _num_samples
from sklearn.model_selection._split import _validate_shuffle_split
from itertools import chain

def multilabel_train_test_split(*arrays,
                                test_size=None,
                                train_size=None,
                                random_state=None,
                                shuffle=True,
                                stratify=None):
    """
    Train test split for multilabel classification. Uses the algorithm from: 
    'Sechidis K., Tsoumakas G., Vlahavas I. (2011) On the Stratification of Multi-Label Data'.
    """
    if stratify is None:
        return train_test_split(*arrays, test_size=test_size,train_size=train_size,
                                random_state=random_state, stratify=None, shuffle=shuffle)
    
    assert shuffle, "Stratified train/test split is not implemented for shuffle=False"
    
    n_arrays = len(arrays)
    arrays = indexable(*arrays)
    n_samples = _num_samples(arrays[0])
    n_train, n_test = _validate_shuffle_split(
        n_samples, test_size, train_size, default_test_size=0.25
    )
    cv = MultilabelStratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=123)
    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
$\endgroup$
2
$\begingroup$

There is a seperate module for classes stratification and no one is going to suggest you to use the train_test_split for this. This could be achieved as follows:

from sklearn.model_selection import StratifiedKFold


train_all = []
evaluate_all = []
skf = StratifiedKFold(n_splits=cv_total, random_state=1234, shuffle=True)
for train_index, evaluate_index in skf.split(train_df.index.values, train_df.coverage_class):
    train_all.append(train_index)
    evaluate_all.append(evaluate_index)
    print(train_index.shape,evaluate_index.shape) # the shape is slightly different in different cv, it's OK

# Getting each batch
def get_cv_data(cv_index):
    train_index = train_all[cv_index-1]
    evaluate_index = evaluate_all[cv_index-1]
    x_train = np.array(train_df.images[train_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    y_train = np.array(train_df.masks[train_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    x_valid = np.array(train_df.images[evaluate_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    y_valid = np.array(train_df.masks[evaluate_index].map(upsample).tolist()).reshape(-1, img_size_target, img_size_target, 1)
    return x_train,y_train,x_valid,y_valid

# Training loop
for cv_index in range(cv_total):
    x_train, y_train, x_valid, y_valid =  get_cv_data(cv_index+1)
    history = model.fit(x_train, y_train,
                        validation_data=[x_valid, y_valid], 
                        epochs=epochs)

This is a simple code snippet for using StratifiedKFold with your code. Just replace the required parameters and hyper-parameters accordingly.

$\endgroup$
1
$\begingroup$

This is because one of your class has only one record. train_test_split is unable to decide where to put that in train or test part. you can do either of following:

  1. you create one/more copy of that record or
  2. remove from the main data or
  3. keep that in a variable and then remove it, then do train_test_split and append it to training data
$\endgroup$
5
  • $\begingroup$ It's foundationally wrong to create copies bc it would lead to data leaks. You would essentially be training on your testing data. $\endgroup$
    – Ari
    Commented Jan 12 at 14:52
  • $\begingroup$ @Ari To avoid test train ambiguity onsame data, first split then make a copy. in same set of test or train it is ok because anyway you need multiple iterations on same data. $\endgroup$
    – Ansh
    Commented Jan 13 at 13:58
  • $\begingroup$ Is there any evidence to back this claim? I.e. augmenting your training with copies of the same data just to meet a stratification goal in numbers? It almost looks like cheating from a statistical learning perspective. $\endgroup$
    – Ari
    Commented Jan 13 at 19:04
  • $\begingroup$ its not the best case but its trying to extrapolate data. even if you are not making a copy.. if not in one iteration but training in different epoch will eventually see same data many of the time. it's very general thing. please check concept of epoch in model training. $\endgroup$
    – Ansh
    Commented Jan 16 at 12:02
  • $\begingroup$ So what you're proposing is identical to having sample weights, datascience.stackexchange.com/a/76449/105175 $\endgroup$
    – Ari
    Commented Jan 16 at 12:19
1
$\begingroup$

If you are using , you could try this:

  • Explode on y -> e.g.
 df.explode('labels')
  • Perform regular train_test_split with stratification
  • Regroup on 'cleaned_text' & aggregate over 'labels' -> e.g.
 df.groupby('cleaned_text').agg({'labels': 'sum')
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.