I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me. so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC. so what do you suggest?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Useful metrics in such scenario are:
- F1 Score (and precision / recall)
- ROC Curves (Metric is : Area Under the ROC Curve (AUC))
Few articles on how to choose metrics for a specific project are:
Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,
What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,
Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.
answered Apr 10 '19 at 12:35
-
1$\begingroup$ I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model. $\endgroup$ – Juan Esteban de la Calle Apr 12 '19 at 17:32
-
$\begingroup$ Yes, AUC is a better description of metric. $\endgroup$ – Shamit Verma Apr 13 '19 at 3:47
$\begingroup$
$\endgroup$
There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.
answered Apr 12 '19 at 6:31
