1
$\begingroup$

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC. so what do you suggest?

Improve this question
$\endgroup$
4
$\begingroup$

Useful metrics in such scenario are:

  • F1 Score (and precision / recall)
  • ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model. $\endgroup$ – Juan Esteban de la Calle Apr 12 '19 at 17:32
  • $\begingroup$ Yes, AUC is a better description of metric. $\endgroup$ – Shamit Verma Apr 13 '19 at 3:47
0
$\begingroup$

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.