Comparing multi-class results with binary classification results

We used machine learning to discriminate the following five disease classes:

• Normal (N)
• Myocardial Infarction (MI)
• Congestive Heart Failure (CHF).

In the past, these diseases were only discriminated with binary classifiers: N and MI, N and CAD, N and CHF.

What is the best way of comparing the multi-class results with the three individual binary classification results?

• Did you mean four classes? N, MI, CAD, CHF . That's what I assumed writing the answer. May 29 '19 at 17:53

(As everything in life) It depends.

Often, a single person (a sample) can be within more than a single class. You do have a set of 3 binary classifiers that are known, these are understood as three models to which we can ask the questions:

• classifier 1, do this person have MI?
• classifier 2, do this person have CAD?
• classifier 3, do this person have CHF?

What is not obvious from the above is that (ignoring any physical constraints - I'm no expect in heart diseases) a person may have MI, CAD and CHF at once. Or just two of them at once, or only one, or none of them.

If you build a classifier that will say that a person has: MI OR CAD OR CHF OR nothing (N); then the binary classifiers cannot be compared against that new classifier because the three classifiers produce a different set of results from the multiclass classifier. But not all is lost, let's work a couple of options that can be explored:

There are physical limitations that allow us to assume that the classes are disjoint

If one can argue that it is physically impossible for someone to have two of: MI, CAD or CHF at once then we can compare the classifiers. Note that this is an assumption that needs a proof outside of the ML process: e.g. the definition of the diseases may exclude the possibility of having them at once.

In this case we need to deal with the case where multiple binary classifiers may output a positive class for the same sample. For example if both the MI and CAD classifiers outputs the positive class for a sample then we need to select either MI or CAD as the true classification. Most binary classifiers can output (or at least estimate) class probabilities, one can use these probabilities to decide for the winning classifier (the output with the higher probability).

(Note: If the classifiers are different model types from each other it is likely that they will require probability calibration but this is beyond this answer.)

With this process you then have a simple set of 4 classes (N, MI, CAD, CHF) for all samples in the dataset. You can use this set to compare against the output of the multiclass classifier.

It is (physically) valid for a person/sample to be in multiple classes at once

Finding the samples for each of these classes is likely to be difficult (and it is likely you will not find any samples at all for some of them). Moreover you will need to rebuild your model to include more classes.

On the other hand this is the perfect case for the use of a multiclass and multilabel classifier. Instead of creating the eight classes (which will likely be unbalanced) several models can predict multiple classes at once. Notably Decision Tress (and forests) and Neural Nets are good at multiclass with multilabel classification.

If you can go for the multilabel classification you will have matrices as output of your model, for example for some 6 samples you may get: