I am working on a classification problem using algorithms such as Logistic Regression, Support Vector Machines, Decision Trees, Random Forests and Naive Bayes. My data consists of binary class classification i.e. Fire(1) or No Fire(0). Due to the imbalance in data Cohen Kappa was recommended for evaluation of model performance.

I am using the scikit-learn sklearn.metrics.cohen_kappa_score to compute the cohen kappa score. To compute the value it takes the following inputs

from sklearn.metrics import cohen_kappa_score
cohen_score = cohen_kappa_score(y_test, predictions)

So it takes the y_test and predictions made by the specific model the same inputs used for confusion matrix and classification report.

However, Cohen Kappa is suppose used to measure the inter-rater agreement between observers or annotators. If we were measuring the quality of a document based on scores by 2 reviewers. Those 2 reviewers would be the annotator. But in the case of a classification problem. When I compute the score in scikit learn as mentioned above. What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?


1 Answer 1


What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?

The two annotators are exactly the two parameters you fed the method with: cohen_kappa_score(y_test, predictions), i.e. the truth/labels and your predictions. And these are then measured in terms of their "inter-rater" agreement.

  • $\begingroup$ Hi Sammy, thanks for answering this. So the annotators are the correct labels and the predictions made by the specific model. This is for my research thesis so I was worried they could ask me this question and I didn't have the answer. $\endgroup$ Apr 4, 2020 at 13:09
  • $\begingroup$ @UsmanAli exactly! Did not find a good source unfortunately but in the following paper the authors state "a measure widely used in other fields as [...] machine learning, where it is used to evaluate the agreement between the actual and the assigned classes by a classifier" (see journals.plos.org/plosone/article?id=10.1371/…) $\endgroup$
    – Jonathan
    Apr 4, 2020 at 13:43
  • $\begingroup$ Thanks for all your help. I have marked your answer as correct now. $\endgroup$ Apr 4, 2020 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.