Binary classification: is there a tradeoff between predicting class 1 vs 0?

Question

I have been digging much more in detail into classification performance metrics lately to get my head around the 'dynamics' of classification algorithms. What I have noticed is that in binary classification problems, I have so far NOT seen an algorithm that does well predicting both classes at the same time.

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

The actual performance metric I calculate at in practice are: (% accuracy Class A-20%) and (% accuracy Class B-80%) for the example of a 20/80 distribution.(% accuracy Class A)=(# corrrecly predicted points A/# points A). Any value of this metric above 0 means the model has an 'edge' predicting that class.

However, what I have noticed is that it is extremely rare that I find values above 0 for both classes. Most of the time (if not always), when there is an edge in a class, there is a somewhat equal or larger loss in the other class.

Formally, taking the sum of (% accuracy of Class A minus 20%)+(% accuracy of Class B minus 80%) results usually in 0 or less. This means you 'gain power' predicting a class, but you lose that power, or more, for the other class. Overall I like this metric because it beats accuracy in case of a majority class 'vote' for a biased model. In case of the majority class bias, this metric would return 0 (try it out you will see, it will end up being -20%+20%=0)

Anything above 0 would be a good model to me. I then thought meta ensembles are the natural progression of this, where u 'just' need to combine the powers of models that have an edge on each class. Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

Am I misinterpreting/understanding anythere here? Is this normal behaviour for binary classifications? Is it not possible to be able to predict both classes above their distribution in the dataset?

Thanks a lot for your enlightments.

P.S: In scikit-learn i always use class_weight='balanced' to avoid balancing/majority class issues etc.

Answer 1

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

Consider the following. Your 80% class is 0. You predict a constant 0 for all datapoints. You have 80% accuracy already with no modelling done and 100% accuracy on the majority class. If you fit a model to that data, you'll get an accuracy above 80% on the majority class and lesser accuracy on the minority class, and that's not a good performance which debunks your theory of a good metric.

The case above is from personal experience, seeing your results makes me think if your datapoints are really separable. It is possible to have accuracies above the distribution marks if your features are good enough for model to use to separate each class from the other which in reality is feasable.

And again the choice of the metric in binary classification will always be a business problem, what is the cost of misclassifying a 1 as a 0 and a 0 as a 1? If the answer to these two questions is not the same, you should have some specific metrics for it.

Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

It really depends on how diverse your base models are. But in general, meta-learners made out of weak learners outperform a single complex model. So a sort of a yes to your question.