# Comparing multi-class vs. binary classifiers in predicting a single class

I've pretty much read the majority of similar questions, but I haven't yet found the answer to my question.

Let's say we have n samples of four different labels/classes namely A, B, C, and D. We train two classifiers:

• First classifier: we train a multi-class classifier to classify a sample in data to one of four classes. Let's say the accuracy of the model is %x.
• Second classifier: now let's say all we care about is that if a sample is A or not A. And we train a binary classifier for classifying samples to either A or non-A. Let's say the accuracy of this models is %y.

My question is, can we compare x and y as a way to measure the performance of classifiers on classifying A? In other words, does a high performance in a multi-class classifier mean that the classifier is capable of recognizing the single classes with high performance as well?

The real-world example of this is that I've read papers that trained multi-class classifiers on a dataset that contains four different types of text. They achieved pretty high performance. But all I care about is for a model to be able to correctly classify one specific type of text. I trained a binary classifier that achieves a lower accuracy. Does this show that my model is working poorly on that type of text and the multi-class classifier is doing better? Or shouldn't I compare these two?

In general we can't compare the performance of a multiclass classifier with the performance of a binary classifier since the former expresses how good the classifier is at classifying any instance of any class. So if there are $$n_A$$ samples labelled A, there's only a proportion of $$n_A/n$$ of the global accuracy of the multiclass classifier which is about A. In particular a multiclass classifier usually tends to favor the largest classes, so if class A happens to be a small proportion of the data then the global performance will not reflect how good it is at classifying A: for example it might have 90% accuracy simply because class B is 90% of the data, this doesn't prove anything about class A. By contrast the performance of the binary classifier is by definition solely about class A.