I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC. so what do you suggest?


2 Answers 2


Useful metrics in such scenario are:

  • F1 Score (and precision / recall)
  • ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

  • 1
    $\begingroup$ I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model. $\endgroup$ Apr 12, 2019 at 17:32
  • $\begingroup$ Yes, AUC is a better description of metric. $\endgroup$ Apr 13, 2019 at 3:47

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.