Catboost multiclassification evaluation metric: Kappa & WKappa

Question

I am working on an unbalanced classification problem and i want to use Kappa as my evaluation metric. Considering the classifier accepts weights (which i have given it), should i still be using weighted kappa or just use the standard kappa? I am not entirely sure of the difference to be honest.

model = CatBoostClassifier(
    iterations = 1000,
    learning_rate = 0.1,
    random_seed = 400,
    one_hot_max_size = 15,
    loss_function = 'MultiClass',
    eval_metric = 'WKappa', # weighted kappa
    #ignored_features = ignore_list,
    class_weights= weights,
    od_type = "Iter",
    od_wait = 50,
    task_type = 'GPU',
    border_count = 124,
    depth = 12
)

Answer 1

I posted the same question on the Catboost github (issues) page and got an answer. The link can be found here: https://github.com/catboost/catboost/issues/1447

Answer:

Class weights and weights in WKappa are different.

Calculation of WKappa metric consists of two steps:

1. Confusion matrix calculation - here object and class weights are used.
2. WKappa calculation using confusion matrix - here values from confusion matrix multiplied by squared difference between classes. (i-j)^2. In other words, weight of class 0 has the same effect when object of class 0 was classified as class 1 and when classified as class 5. But WKappa weight for misclassification 0->5 will be 25 times more than for misclassification 0->1. So you can use both class weights and WKappa if your classes are ordinal.