accuracy, precision, f1, ROC are good for binary single class problem.
but for more complex problem (imbalance multi-class problem), what should i use?
Do you have any recommendation?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
One standard metric is the top-1 or top-5 test error rate. For instance, for top-5, your model predict 5 most likely labels, and if none of the 5 labels is the ground truth label, you mark this instance as an error. This is usually a standard metric when people working with the ImageNet data. See example here usage here. This metric does not explicitly count for that the classes are imbalanced.
Another powerful metric is the Mean Average Precision (mAP), where you calculate the Average Precisoin (AP) for each class and then average among all classes. In this metric the smaller classes get equal weights compared to larger classes. The idea comes from the information retrieval community. This metric is also suitable for multi-label classification. See example usage here.
