I have been digging much more in detail into classification performance metrics lately to get my head around the 'dynamics' of classification algorithms. What I have noticed is that in binary classification problems, I have so far NOT seen an algorithm that does well predicting both classes at the same time.
To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.
The actual performance metric I calculate at in practice are: (% accuracy Class A-20%) and (% accuracy Class B-80%) for the example of a 20/80 distribution.(% accuracy Class A)=(# corrrecly predicted points A/# points A). Any value of this metric above 0 means the model has an 'edge' predicting that class.
However, what I have noticed is that it is extremely rare that I find values above 0 for both classes. Most of the time (if not always), when there is an edge in a class, there is a somewhat equal or larger loss in the other class.
Formally, taking the sum of (% accuracy of Class A minus 20%)+(% accuracy of Class B minus 80%) results usually in 0 or less. This means you 'gain power' predicting a class, but you lose that power, or more, for the other class. Overall I like this metric because it beats accuracy in case of a majority class 'vote' for a biased model. In case of the majority class bias, this metric would return 0 (try it out you will see, it will end up being -20%+20%=0)
Anything above 0 would be a good model to me. I then thought meta ensembles are the natural progression of this, where u 'just' need to combine the powers of models that have an edge on each class. Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.
Am I misinterpreting/understanding anythere here? Is this normal behaviour for binary classifications? Is it not possible to be able to predict both classes above their distribution in the dataset?
Thanks a lot for your enlightments.
P.S: In scikit-learn i always use class_weight='balanced' to avoid balancing/majority class issues etc.