GridSearchCV for lightbgm classifier for multiclass problem

Question

I am doing the following:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score, train_test_split    
import lightgbm as lgb


param_test ={
                 'learning_rate' : [0.01, 0.02, 0.03, 0.04, 0.05, 0.08, 0.1, 0.2, 0.3, 0.4]
                }


clf = lgb.LGBMClassifier(boosting_type='gbdt',\
                   num_leaves=31, \
                   max_depth=-1, \
                   n_estimators=100, \
                   subsample_for_bin=200000, \
                   objective='multiclass', \
                   class_weight=balanced, \
                   min_split_gain=0.0, \
                   min_child_weight=0.001, \
                   min_child_samples=20, \
                   subsample=1.0, \
                   subsample_freq=0, \
                   colsample_bytree=1.0, \
                   reg_alpha=0.0, \
                   reg_lambda=0.0, \
                   random_state=None,\
                   n_jobs=-1,\
                   silent=True, \
                   importance_type='split'
                  )


gs = GridSearchCV(
                    estimator=clf,
                    param_grid = param_test, 
                    scoring='roc_auc',
                    cv=3
                  )

gs.fit(X_train, y_train_lbl["target_encoded"].values)

and I am getting the below error:

    /home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _score(estimator, X_test, y_test, scorer, is_multimetric)
    597     """
    598     if is_multimetric:
--> 599         return _multimetric_score(estimator, X_test, y_test, scorer)
    600     else:
    601         if y_test is None:

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _multimetric_score(estimator, X_test, y_test, scorers)
    627             score = scorer(estimator, X_test)
    628         else:
--> 629             score = scorer(estimator, X_test, y_test)
    630 
    631         if hasattr(score, 'item'):

/home/cdsw/.local/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    173         y_type = type_of_target(y)
    174         if y_type not in ("binary", "multilabel-indicator"):
--> 175             raise ValueError("{0} format is not supported".format(y_type))
    176 
    177         if is_regressor(clf):

**ValueError: multiclass format is not supported**

So, the value error for the multiclass not being supported is what has stumped me. Am I missing some fundamentals here? I used auc as a metric. Should this be multi_logloss? I tried that as well no result.

Answer 1

roc_auc can not be used as a metric for multiclass models in scikit-learn, only for binary classifiers or one-vs-rest classifiers. Scikit-learn's document discusses it here.

Answer 2

I cannot run the code, but I guess it's because scoring you've chosen, in particular, roc_auc. This metric/loss function is only for binary classification while you have a multiclass problem. You can try just accuracy_score, but it works bad when classes have different ratios in dataset. People on Kaggle very often use MultiClass Log Loss for this kind of problems. Here's the code, I found it here.

import numpy as np

def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    """Multi class version of Logarithmic Loss metric.
    https://www.kaggle.com/wiki/MultiClassLogLoss

    idea from this post:
    http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209

    Parameters
    ----------
    y_true : array, shape = [n_samples]
    y_pred : array, shape = [n_samples, n_classes]

    Returns
    -------
    loss : float
    """
    predictions = np.clip(y_pred, eps, 1 - eps)

    # normalize row sums to 1
    predictions /= predictions.sum(axis=1)[:, np.newaxis]

    actual = np.zeros(y_pred.shape)
    rows = actual.shape[0]
    actual[np.arange(rows), y_true.astype(int)] = 1
    vsota = np.sum(actual * np.log(predictions))
    return -1.0 / rows * vsota

However, I guess for GridSearchCV in sklearn it's not enough. You can use custom scorers like function above, but you need to add make_scorer decorator:

NOTE that when using custom scorers, each scorer should return a single value. Metric functions returning a list/array of values can be wrapped into multiple scorers that return one value each. (from sklearn documentation)

from sklearn.metrics import make_scorer


@make_scorer
def multiclass_log_loss(y_true, y_pred, eps=1e-15):
    # function from snippet above

Pay attention to model outputs and inputs of this function, shape should be the same. Also, I would recommend reading GridSearchCV documentation - it may help as well.