2
$\begingroup$

I am working on an unbalanced classification problem and i want to use Kappa as my evaluation metric. Considering the classifier accepts weights (which i have given it), should i still be using weighted kappa or just use the standard kappa? I am not entirely sure of the difference to be honest.

model = CatBoostClassifier(
    iterations = 1000,
    learning_rate = 0.1,
    random_seed = 400,
    one_hot_max_size = 15,
    loss_function = 'MultiClass',
    eval_metric = 'WKappa', # weighted kappa
    #ignored_features = ignore_list,
    class_weights= weights,
    od_type = "Iter",
    od_wait = 50,
    task_type = 'GPU',
    border_count = 124,
    depth = 12
)
$\endgroup$
1
$\begingroup$

I posted the same question on the Catboost github (issues) page and got an answer. The link can be found here: https://github.com/catboost/catboost/issues/1447

Answer:

Class weights and weights in WKappa are different.

Calculation of WKappa metric consists of two steps:

  1. Confusion matrix calculation - here object and class weights are used.
  2. WKappa calculation using confusion matrix - here values from confusion matrix multiplied by squared difference between classes. (i-j)^2. In other words, weight of class 0 has the same effect when object of class 0 was classified as class 1 and when classified as class 5. But WKappa weight for misclassification 0->5 will be 25 times more than for misclassification 0->1. So you can use both class weights and WKappa if your classes are ordinal.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.