1
$\begingroup$

Let's consider an example. I have patient level data, their symptoms, reading from various medical tests. Based on that, I have built a binary classifier given patient data to classify if they are likely to have a disease, and if so, I'd want to manually run medical tests on them. For that case, we look at the disease class and rank order the predicted probabilities and select the riskiest patients to have the disease.

A few months later, say the disease evolves into a different strain. So we already have an imbalanced class of healthy vs infected patients. Not the new strain will be still a smaller population. Apart from that, the symptoms, various test readings will also be different. So translating this to a multi-classification problem makes sense. When you build a multi-classifier, and you have three probabilities, how would you go about rank ordering the same set of patients?

If you're interested in the risk of patients likely to be infected, taking the max of the either of the strain of disease a good way to proceed or is there a method to choose them based on their individual probabilities?

Edit: To bring a little more clarity. The multi:softprob gives the probability for each of the classes. If you have three classes, it will give three probabilities for each class summing up to 1. More details are here.

Given that we will get multiple probabilities for each row/patient from the example above, how do you go about choosing the final probability to rank order risky patients.

$\endgroup$
0
$\begingroup$

Multiclassification with xgboost is normally done One vs All. Meaning that first you try to predict if it belongs to class one or not. Then two second class or not...

You can understand as the probability of belongin to one class and not to the rest.

Let's put an example for better understanding: say that you have a ton of patients that have 3 illnes: Corona Virus, Sars and HIV.

You build your multiclassification model with xgboost or whatever and for a new patient you do a prediction and check the probabilities:

Corona [0.8] Sars [0.1] HIV [0.01] Not sick [0.09]

What you do from here is select the most probable cause (corona) and that is your final class.

If you do multilabel you can get something like this too Corona [0.9] Sars [0.20] HIV [0.999]

Meaning that even if it has a high probability of getting corona it also has a higher probability of having HIV. In multiclassification is different from multilabel where you can have a patient that has both of the sickness

From the sklearn documentation:

The one-vs-the-rest meta-classifier also implements a predict_proba method, so long as such a method is implemented by the base classifier. This method returns probabilities of class membership in both the single label and multilabel case. Note that in the multilabel case, probabilities are the marginal probability that a given sample falls in the given class. As such, in the multilabel case the sum of these probabilities over all possible labels for a given sample will not sum to unity, as they do in the single label case.

In summary, if sickness are independent it will predict the probability of the patient having one sickness, there has to be a class not sick just if there is the possibility for the patient not being sick

Note: In NN is different

| improve this answer | |
$\endgroup$
  • $\begingroup$ The probabilities of each class should add up to one right? So the above values should not occur. $\endgroup$ – Next Door Engineer Mar 16 at 11:34
  • $\begingroup$ You are right!! Corrected $\endgroup$ – Carlos Mougan Mar 16 at 13:46
  • $\begingroup$ My understanding is that xgboost uses one-v-rest together with softmax, so that the output "probability"s should indeed sum to one. (?) $\endgroup$ – Ben Reiniger Mar 16 at 14:12
  • $\begingroup$ @CarlosMougan I've added details with multi:softprob. The example you gave is an extreme situation where you have the probability of Corona at 0.8. What happens when the probabilities are Corona [0.4] Sars [0.1] HIV [0.01] Not sick [0.49] or Corona [0.31] Sars [0.3] HIV [0.29] Not sick [0.1]. At this point, which probability should take? Do you take the max of the diseases to get riskiest patients or max of all? If you take max of all, won't the healthy patients have a bias? $\endgroup$ – Next Door Engineer Mar 16 at 14:31
  • $\begingroup$ The algorithm by default will take the highest one, but you can always change it to the one that suits you better. Per example, if no class is higher than 0.8 classify as other. You could then optimize the metric instead $\endgroup$ – Carlos Mougan Mar 17 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.