-1
$\begingroup$

I am doing the following:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score, train_test_split    
import lightgbm as lgb


param_test ={
                 'learning_rate' : [0.01, 0.02, 0.03, 0.04, 0.05, 0.08, 0.1, 0.2, 0.3, 0.4]
                }


clf = lgb.LGBMClassifier(boosting_type='gbdt',\
                   num_leaves=31, \
                   max_depth=-1, \
                   n_estimators=100, \
                   subsample_for_bin=200000, \
                   objective='multiclass', \
                   class_weight=balanced, \
                   min_split_gain=0.0, \
                   min_child_weight=0.001, \
                   min_child_samples=20, \
                   subsample=1.0, \
                   subsample_freq=0, \
                   colsample_bytree=1.0, \
                   reg_alpha=0.0, \
                   reg_lambda=0.0, \
                   random_state=None,\
                   n_jobs=-1,\
                   silent=True, \
                   importance_type='split'
                  )


gs = GridSearchCV(
                    estimator=clf,
                    param_grid = param_test, 
                    scoring='roc_auc',
                    cv=3
                  )

gs.fit(X_train, y_train_lbl["target_encoded"].values)

and I am getting the below error:

    /home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _score(estimator, X_test, y_test, scorer, is_multimetric)
    597     """
    598     if is_multimetric:
--> 599         return _multimetric_score(estimator, X_test, y_test, scorer)
    600     else:
    601         if y_test is None:

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _multimetric_score(estimator, X_test, y_test, scorers)
    627             score = scorer(estimator, X_test)
    628         else:
--> 629             score = scorer(estimator, X_test, y_test)
    630 
    631         if hasattr(score, 'item'):

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    173         y_type = type_of_target(y)
    174         if y_type not in ("binary", "multilabel-indicator"):
--> 175             raise ValueError("{0} format is not supported".format(y_type))
    176 
    177         if is_regressor(clf):

**ValueError: multiclass format is not supported**

So, the value error for the multiclass not being supported is what has stumped me. Am I missing some fundamentals here? I used auc as a metric. Should this be multi_logloss? I tried that as well no result.

$\endgroup$
2
  • $\begingroup$ A first guess: you need to binarize (dummy-encode) the target? The last call in the error trace includes "multilabel-indicator" as an acceptable mode. $\endgroup$ – Ben Reiniger Nov 6 '19 at 14:47
  • $\begingroup$ Please, provide a sample of the data you're using. This: **ValueError: multiclass format is not supported** suggests that the target variable is in a format which can't be fed to the model. $\endgroup$ – 89f3a1c Nov 6 '19 at 15:40
1
$\begingroup$

roc_auc can not be used as a metric for multiclass models in scikit-learn, only for binary classifiers or one-vs-rest classifiers. Scikit-learn's document discusses it here.

$\endgroup$
4
  • $\begingroup$ then what should I use $\endgroup$ – Sachin Yadav Nov 6 '19 at 15:57
  • $\begingroup$ You can use zero-one loss, accuracy, recall, precision, F score, log loss, or define your custom metric. $\endgroup$ – Brian Spiering Nov 6 '19 at 16:54
  • 1
    $\begingroup$ Well, turns out OP not only plagiarized your answer word by word (including the comment!) in an SO thread (you can't see his answer now, it was deleted after being flagged for plagiarism), not only his post here is identical to the SO one, but he was not even grateful enough to accept and upvote your answer, while there he was probing the OP to accept (he even got 3 upvotes!). $\endgroup$ – desertnaut Nov 11 '19 at 17:52
  • $\begingroup$ Anyway, sole upvote here is from me, but I'm afraid this question will also be deleted (I have raised a flag)... :( $\endgroup$ – desertnaut Nov 11 '19 at 17:52
0
$\begingroup$

I cannot run the code, but I guess it's because scoring you've chosen, in particular, roc_auc. This metric/loss function is only for binary classification while you have a multiclass problem. You can try just accuracy_score, but it works bad when classes have different ratios in dataset. People on Kaggle very often use MultiClass Log Loss for this kind of problems. Here's the code, I found it here.

import numpy as np

def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    """Multi class version of Logarithmic Loss metric.
    https://www.kaggle.com/wiki/MultiClassLogLoss

    idea from this post:
    http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209

    Parameters
    ----------
    y_true : array, shape = [n_samples]
    y_pred : array, shape = [n_samples, n_classes]

    Returns
    -------
    loss : float
    """
    predictions = np.clip(y_pred, eps, 1 - eps)

    # normalize row sums to 1
    predictions /= predictions.sum(axis=1)[:, np.newaxis]

    actual = np.zeros(y_pred.shape)
    rows = actual.shape[0]
    actual[np.arange(rows), y_true.astype(int)] = 1
    vsota = np.sum(actual * np.log(predictions))
    return -1.0 / rows * vsota

However, I guess for GridSearchCV in sklearn it's not enough. You can use custom scorers like function above, but you need to add make_scorer decorator:

NOTE that when using custom scorers, each scorer should return a single value. Metric functions returning a list/array of values can be wrapped into multiple scorers that return one value each. (from sklearn documentation)

from sklearn.metrics import make_scorer


@make_scorer
def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    # function from snippet above

Pay attention to model outputs and inputs of this function, shape should be the same. Also, I would recommend reading GridSearchCV documentation - it may help as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.