0
$\begingroup$

I am doing the following:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score, train_test_split    
import lightgbm as lgb


param_test ={
                 'learning_rate' : [0.01, 0.02, 0.03, 0.04, 0.05, 0.08, 0.1, 0.2, 0.3, 0.4]
                }


clf = lgb.LGBMClassifier(boosting_type='gbdt',\
                   num_leaves=31, \
                   max_depth=-1, \
                   n_estimators=100, \
                   subsample_for_bin=200000, \
                   objective='multiclass', \
                   class_weight=balanced, \
                   min_split_gain=0.0, \
                   min_child_weight=0.001, \
                   min_child_samples=20, \
                   subsample=1.0, \
                   subsample_freq=0, \
                   colsample_bytree=1.0, \
                   reg_alpha=0.0, \
                   reg_lambda=0.0, \
                   random_state=None,\
                   n_jobs=-1,\
                   silent=True, \
                   importance_type='split'
                  )


gs = GridSearchCV(
                    estimator=clf,
                    param_grid = param_test, 
                    scoring='roc_auc',
                    cv=3
                  )

gs.fit(X_train, y_train_lbl["target_encoded"].values)

and I am getting the below error:

    /home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _score(estimator, X_test, y_test, scorer, is_multimetric)
    597     """
    598     if is_multimetric:
--> 599         return _multimetric_score(estimator, X_test, y_test, scorer)
    600     else:
    601         if y_test is None:

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _multimetric_score(estimator, X_test, y_test, scorers)
    627             score = scorer(estimator, X_test)
    628         else:
--> 629             score = scorer(estimator, X_test, y_test)
    630 
    631         if hasattr(score, 'item'):

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    173         y_type = type_of_target(y)
    174         if y_type not in ("binary", "multilabel-indicator"):
--> 175             raise ValueError("{0} format is not supported".format(y_type))
    176 
    177         if is_regressor(clf):

**ValueError: multiclass format is not supported**

So, the value error for the multiclass not being supported is what has stumped me. Am I missing some fundamentals here? I used auc as a metric. Should this be multi_logloss? I tried that as well no result.

$\endgroup$
2
  • $\begingroup$ A first guess: you need to binarize (dummy-encode) the target? The last call in the error trace includes "multilabel-indicator" as an acceptable mode. $\endgroup$
    – Ben Reiniger
    Nov 6, 2019 at 14:47
  • $\begingroup$ Please, provide a sample of the data you're using. This: **ValueError: multiclass format is not supported** suggests that the target variable is in a format which can't be fed to the model. $\endgroup$
    – 89f3a1c
    Nov 6, 2019 at 15:40

2 Answers 2

1
$\begingroup$

roc_auc can not be used as a metric for multiclass models in scikit-learn, only for binary classifiers or one-vs-rest classifiers. Scikit-learn's document discusses it here.

$\endgroup$
4
  • $\begingroup$ then what should I use $\endgroup$ Nov 6, 2019 at 15:57
  • $\begingroup$ You can use zero-one loss, accuracy, recall, precision, F score, log loss, or define your custom metric. $\endgroup$ Nov 6, 2019 at 16:54
  • 1
    $\begingroup$ Well, turns out OP not only plagiarized your answer word by word (including the comment!) in an SO thread (you can't see his answer now, it was deleted after being flagged for plagiarism), not only his post here is identical to the SO one, but he was not even grateful enough to accept and upvote your answer, while there he was probing the OP to accept (he even got 3 upvotes!). $\endgroup$
    – desertnaut
    Nov 11, 2019 at 17:52
  • $\begingroup$ Anyway, sole upvote here is from me, but I'm afraid this question will also be deleted (I have raised a flag)... :( $\endgroup$
    – desertnaut
    Nov 11, 2019 at 17:52
0
$\begingroup$

I cannot run the code, but I guess it's because scoring you've chosen, in particular, roc_auc. This metric/loss function is only for binary classification while you have a multiclass problem. You can try just accuracy_score, but it works bad when classes have different ratios in dataset. People on Kaggle very often use MultiClass Log Loss for this kind of problems. Here's the code, I found it here.

import numpy as np

def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    """Multi class version of Logarithmic Loss metric.
    https://www.kaggle.com/wiki/MultiClassLogLoss

    idea from this post:
    http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209

    Parameters
    ----------
    y_true : array, shape = [n_samples]
    y_pred : array, shape = [n_samples, n_classes]

    Returns
    -------
    loss : float
    """
    predictions = np.clip(y_pred, eps, 1 - eps)

    # normalize row sums to 1
    predictions /= predictions.sum(axis=1)[:, np.newaxis]

    actual = np.zeros(y_pred.shape)
    rows = actual.shape[0]
    actual[np.arange(rows), y_true.astype(int)] = 1
    vsota = np.sum(actual * np.log(predictions))
    return -1.0 / rows * vsota

However, I guess for GridSearchCV in sklearn it's not enough. You can use custom scorers like function above, but you need to add make_scorer decorator:

NOTE that when using custom scorers, each scorer should return a single value. Metric functions returning a list/array of values can be wrapped into multiple scorers that return one value each. (from sklearn documentation)

from sklearn.metrics import make_scorer


@make_scorer
def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    # function from snippet above

Pay attention to model outputs and inputs of this function, shape should be the same. Also, I would recommend reading GridSearchCV documentation - it may help as well.

$\endgroup$
1
  • $\begingroup$ How do you use this scorer - by name, or? $\endgroup$
    – jtlz2
    Dec 1, 2022 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.