9
$\begingroup$

I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.

But I couldn't find kappa as an evaluation_metric in scikit learn here sklearn.metrics.

Questions

  1. Is there any workaround for including kappa in gridsearchcv?
  2. Is there any other better metric i can use instead of kappa in scikit learn?
$\endgroup$
  • 1
    $\begingroup$ Fork sklearn and add support yourself; it's not that hard! $\endgroup$ – Emre Sep 16 '15 at 7:31
  • $\begingroup$ K. Are you saying modify the sklearn source file I think griddearchcv.py file? Any directions will be great. $\endgroup$ – GeorgeOfTheRF Sep 16 '15 at 7:42
  • $\begingroup$ Look at the definition of sklearn.metrics.f1_score $\endgroup$ – Emre Sep 16 '15 at 21:18
12
$\begingroup$

Cohen's kappa was introduced in scikit-learn 0.17.

You can wrap it in make_scorer for use in GridSearchCV.

from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC

kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks. What does make_scorer() do? $\endgroup$ – GeorgeOfTheRF Nov 26 '15 at 9:51
  • $\begingroup$ make_scorer() converts metrics into callables that can be used for model evaluation. The scoring argument expects a function scorer(estimator, X, y). So it will fail, if you try to pass scoring=cohen_kappa_score directly, since the signature is different, cohen_kappa_score(y1, y2, labels=None). The make_scorer also accepts additional arguments, like labels from cohen_kappa_score. $\endgroup$ – Tulio Casagrande Nov 26 '15 at 11:58
2
$\begingroup$

In addition to the link in the existing answer, there is also a Scikit-Learn laboratory, where methods and algorithms are being experimented.

In case you are okay with working with bleeding edge code, this library would be a nice reference.

The Cohen's Kappa is also one of the metrics in the library, which takes in true labels, predicted labels, weights and allowing one off? as the input parameters. Obviously, the metric would range from [-1, 1].

You can also have a look at the implementation code, just in case you would like to contribute.

Note: Cohen's Kappa is also implemented in Scikit-Learn.

Yes, there are alternatives to the Cohen Kappa metric.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.