I am working on a classification problem using algorithms such as Logistic Regression, Support Vector Machines, Decision Trees, Random Forests and Naive Bayes. My data consists of binary class classification i.e. Fire(1) or No Fire(0). Due to the imbalance in data Cohen Kappa was recommended for evaluation of model performance.
I am using the scikit-learn sklearn.metrics.cohen_kappa_score to compute the cohen kappa score. To compute the value it takes the following inputs
from sklearn.metrics import cohen_kappa_score
cohen_score = cohen_kappa_score(y_test, predictions)
print(cohen_score)
So it takes the y_test and predictions made by the specific model the same inputs used for confusion matrix and classification report.
However, Cohen Kappa is suppose used to measure the inter-rater agreement between observers or annotators. If we were measuring the quality of a document based on scores by 2 reviewers. Those 2 reviewers would be the annotator. But in the case of a classification problem. When I compute the score in scikit learn as mentioned above. What 2 annotators or observers does Cohen Kappa use in classification problems when you compute the score in scikit learn?
