I've pretty much read the majority of similar questions, but I haven't yet found the answer to my question.
Let's say we have n
samples of four different labels/classes namely A
, B
, C
, and D
. We train two classifiers:
First classifier
: we train a multi-class classifier to classify a sample in data to one of four classes. Let's say the accuracy of the model is %x.Second classifier
: now let's say all we care about is that if a sample is A or not A. And we train a binary classifier for classifying samples to either A or non-A. Let's say the accuracy of this models is %y.
My question is, can we compare x and y as a way to measure the performance of classifiers on classifying A? In other words, does a high performance in a multi-class classifier mean that the classifier is capable of recognizing the single classes with high performance as well?
The real-world example of this is that I've read papers that trained multi-class classifiers on a dataset that contains four different types of text. They achieved pretty high performance. But all I care about is for a model to be able to correctly classify one specific type of text. I trained a binary classifier that achieves a lower accuracy. Does this show that my model is working poorly on that type of text and the multi-class classifier is doing better? Or shouldn't I compare these two?