Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have class imbalance in the ratio 1:15 i.e. very low event rate. So to select tuning parameters of GBM in scikit learn I want to use Kappa instead of F1 score. My understanding is Kappa is a better metric than F1 score for class imbalance.
But I couldn't find kappa as an evaluation_metric in scikit learn here sklearn.metrics.
Questions
Cohen's kappa was introduced in scikit-learn 0.17.
You can wrap it in make_scorer for use in GridSearchCV.
from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC
kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)
make_scorer() converts metrics into callables that can be used for model evaluation. The scoring argument expects a function scorer(estimator, X, y). So it will fail, if you try to pass scoring=cohen_kappa_score directly, since the signature is different, cohen_kappa_score(y1, y2, labels=None). The make_scorer also accepts additional arguments, like labels from cohen_kappa_score.
$\endgroup$
Nov 26, 2015 at 11:58
In addition to the link in the existing answer, there is also a Scikit-Learn laboratory, where methods and algorithms are being experimented.
In case you are okay with working with bleeding edge code, this library would be a nice reference.
The Cohen's Kappa is also one of the metrics in the library, which takes in true labels, predicted labels, weights and allowing one off? as the input parameters. Obviously, the metric would range from [-1, 1].
You can also have a look at the implementation code, just in case you would like to contribute.
Note: Cohen's Kappa is also implemented in Scikit-Learn.
Yes, there are alternatives to the Cohen Kappa metric.
