We used machine learning to discriminate the following five disease classes:

  • Normal (N)
  • Myocardial Infarction (MI)
  • Coronary Artery Disease (CAD)
  • Congestive Heart Failure (CHF).

In the past, these diseases were only discriminated with binary classifiers: N and MI, N and CAD, N and CHF.

What is the best way of comparing the multi-class results with the three individual binary classification results?

Access to the confusion matrix is available for all cases.

| improve this question | | | | |
  • $\begingroup$ Did you mean four classes? N, MI, CAD, CHF . That's what I assumed writing the answer. $\endgroup$ – grochmal May 29 '19 at 17:53

(As everything in life) It depends.

Often, a single person (a sample) can be within more than a single class. You do have a set of 3 binary classifiers that are known, these are understood as three models to which we can ask the questions:

  • classifier 1, do this person have MI?
  • classifier 2, do this person have CAD?
  • classifier 3, do this person have CHF?

What is not obvious from the above is that (ignoring any physical constraints - I'm no expect in heart diseases) a person may have MI, CAD and CHF at once. Or just two of them at once, or only one, or none of them.

If you build a classifier that will say that a person has: MI OR CAD OR CHF OR nothing (N); then the binary classifiers cannot be compared against that new classifier because the three classifiers produce a different set of results from the multiclass classifier. But not all is lost, let's work a couple of options that can be explored:

There are physical limitations that allow us to assume that the classes are disjoint

If one can argue that it is physically impossible for someone to have two of: MI, CAD or CHF at once then we can compare the classifiers. Note that this is an assumption that needs a proof outside of the ML process: e.g. the definition of the diseases may exclude the possibility of having them at once.

In this case we need to deal with the case where multiple binary classifiers may output a positive class for the same sample. For example if both the MI and CAD classifiers outputs the positive class for a sample then we need to select either MI or CAD as the true classification. Most binary classifiers can output (or at least estimate) class probabilities, one can use these probabilities to decide for the winning classifier (the output with the higher probability).

(Note: If the classifiers are different model types from each other it is likely that they will require probability calibration but this is beyond this answer.)

With this process you then have a simple set of 4 classes (N, MI, CAD, CHF) for all samples in the dataset. You can use this set to compare against the output of the multiclass classifier.

It is (physically) valid for a person/sample to be in multiple classes at once

In this case the multiclass classifier is not classifying four classes, instead it needs to classify eight classes (N, MI, CAD, CHF, MI+CAD, MI+CHF, CAD+CHF, MI+CAD+CHF).

Finding the samples for each of these classes is likely to be difficult (and it is likely you will not find any samples at all for some of them). Moreover you will need to rebuild your model to include more classes.

On the other hand this is the perfect case for the use of a multiclass and multilabel classifier. Instead of creating the eight classes (which will likely be unbalanced) several models can predict multiple classes at once. Notably Decision Tress (and forests) and Neural Nets are good at multiclass with multilabel classification.

If you can go for the multilabel classification you will have matrices as output of your model, for example for some 6 samples you may get:

0   0   0
1   0   0
1   1   0
0   0   1
0   1   1
0   0   0

This can be directly compare against the output of all three binary classifiers on the same 6 samples.

If you go for the 8 classes (and I strongly advise against that - it is always a pain to deal with very unballanced classes) you can directly compare the output of the binary classifiers against the list of results. (e.g. class MI+CAD means that both the 1st and 2nd binary classifiers did output a positive result for a sample).

P.S. Why one would prefer multilabel classifiers over building the classes by hand? Multilabel classifiers allow one to predict multiple classes at once, which do work as a "new class", yet they perform this as part of their decision algorithm. As a rule of thumb a multilabel classifier is less likely to impact the separability of other classes by adding this "new class" as a combination of existing classes. A new class built by hand can affect the separability of the other classes if the algorithm (or hyperparameters) weight classes based on their (low) support (number of elements in class against total number of samples).

| improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.