3
$\begingroup$

Working with Sklearn stratified kfold split, and when I attempt to split using multi-class, I received on error (see below). When I tried and split using binary, it works no problem.

num_classes = len(np.unique(y_train))
y_train_categorical = keras.utils.to_categorical(y_train, num_classes)
kf=StratifiedKFold(n_splits=5, shuffle=True, random_state=999)

splitting data into different folds

for i, (train_index, val_index) in enumerate(kf.split(x_train, y_train_categorical)):
   x_train_kf, x_val_kf = x_train[train_index], x_train[val_index]
   y_train_kf, y_val_kf = y_train[train_index], y_train[val_index]

ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.

Is there a way I can used KFold with multi-class?

$\endgroup$
2
  • $\begingroup$ what is your question, sir? $\endgroup$ Jan 30 '18 at 2:08
  • $\begingroup$ Is there a way I can used KFold with multi-class? $\endgroup$
    – jKraut
    Jan 30 '18 at 2:14
2
$\begingroup$

There is an easier way instead of using loops. Scikit provides cross_val_score.

from sklearn.cross_validation import KFold, cross_val_score
k_fold = KFold(len(y), n_folds=10, shuffle=True, random_state=0)
clf = <any classifier>
print cross_val_score(clf, X, y, cv=k_fold, n_jobs=1)

The topic also has been discussed here.

You can also see here which has a code snippet which may help you:

from sklearn.model_selection import KFold
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4])
kf = KFold(n_splits=2)
kf.get_n_splits(X)

print(kf)  

for train_index, test_index in kf.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

which The first n_samples % n_splits folds have size n_samples // n_splits + 1, other folds have size n_samples // n_splits, where n_samples is the number of samples.

$\endgroup$
4
  • $\begingroup$ thanks for the answer, what if I have imbalanced data and multi label classification. do you have any idea how to split the data to make sure there is enough sample in each label category. it seems kFold Stratified does not support MultiLabel ing. Thanks :) $\endgroup$
    – Maria
    Aug 14 '18 at 5:18
  • $\begingroup$ it raises this error for me indexer = self.loc._convert_to_indexer(key, axis=1) File "C:\Users\saria\Anaconda3\envs\py27\lib\site-packages\pandas\core\indexing.py", line 1327, in _convert_to_indexer .format(mask=objarr[mask])) KeyError: '[ 601 602 603 ... 6005 6006 6007] not in index' do you have any idea of this? $\endgroup$
    – Maria
    Aug 15 '18 at 2:45
  • $\begingroup$ @sariaGoudarzi it highly depends on the real distribution of your data. If the nature is somehow a situation that you can have balanced labels but is not currently, you can employ data augmentation techniques which may differ depending on the data intrinsic. $\endgroup$ Aug 18 '18 at 9:26
  • $\begingroup$ @sariaGoudarzi which line you got that? $\endgroup$ Aug 18 '18 at 9:26
4
$\begingroup$

I had the same problem, you can find my detailed answer here.

Basically, KFold does not recognize your target as multi-class because it relies on these definitions:

  • 'binary': y contains <= 2 discrete values and is 1d or a column vector.
  • 'multiclass': y contains more than two discrete values, is not a sequence of sequences, and is 1d or a column vector.
  • 'multiclass-multioutput': y is a 2d array that contains more than two discrete values, is not a sequence of sequences, and both dimensions are of size > 1.
  • 'multilabel-indicator': y is a label indicator matrix, an array of two dimensions with at least two columns, and at most 2 unique values.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.