Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am training a classifier (e.g, animals), and since many categories are too similar (e.g., insects), I am grouping some categories together (e.g., bugs and mosquitoes as insects). Then, I will train a classifier to distinguish between the different insects.
How can I evaluate the performance of the two together?
Let's say that the first level is ['cat', 'dog', 'insects'] with accuracy of 90%
And the second level is ['mosquito', 'bug'] with accuracy of 80%
What's the overall accuracy?
First mind that accuracy is not a very good performance measure in case there is some imbalance between the classes. Micro or macro F1-score are more informative.
Any overall performance measure can be obtained by taking the full set of instances, considering only the final predicted label against the true label for every instance. From there a confusion matrix can be obtained the usual way. The intermediate level (e.g. "insect") does not matter. In the case of accuracy you just need to count for every instance whether the true label is the same as the final predicted label.
