I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.

But I couldn't find kappa as an evaluation_metric in scikit learn here sklearn.metrics.


  1. Is there any workaround for including kappa in gridsearchcv?
  2. Is there any other better metric i can use instead of kappa in scikit learn?
  • 1
    $\begingroup$ Fork sklearn and add support yourself; it's not that hard! $\endgroup$
    – Emre
    Commented Sep 16, 2015 at 7:31
  • $\begingroup$ K. Are you saying modify the sklearn source file I think griddearchcv.py file? Any directions will be great. $\endgroup$ Commented Sep 16, 2015 at 7:42
  • $\begingroup$ Look at the definition of sklearn.metrics.f1_score $\endgroup$
    – Emre
    Commented Sep 16, 2015 at 21:18

2 Answers 2


Cohen's kappa was introduced in scikit-learn 0.17.

You can wrap it in make_scorer for use in GridSearchCV.

from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC

kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)
  • $\begingroup$ make_scorer() converts metrics into callables that can be used for model evaluation. The scoring argument expects a function scorer(estimator, X, y). So it will fail, if you try to pass scoring=cohen_kappa_score directly, since the signature is different, cohen_kappa_score(y1, y2, labels=None). The make_scorer also accepts additional arguments, like labels from cohen_kappa_score. $\endgroup$ Commented Nov 26, 2015 at 11:58

In addition to the link in the existing answer, there is also a Scikit-Learn laboratory, where methods and algorithms are being experimented.

In case you are okay with working with bleeding edge code, this library would be a nice reference.

The Cohen's Kappa is also one of the metrics in the library, which takes in true labels, predicted labels, weights and allowing one off? as the input parameters. Obviously, the metric would range from [-1, 1].

You can also have a look at the implementation code, just in case you would like to contribute.

Note: Cohen's Kappa is also implemented in Scikit-Learn.

Yes, there are alternatives to the Cohen Kappa metric.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.