0
$\begingroup$

I have a C5.0 model that is trained to predict binary class (C1/C2) on a dataset with 20 features. The model is configured to perform boosting (10 trials) and it has a miss-classification cost function (100:1 where 100 is the cost for miss-classifying a Negative sample as Positive and 1 is the cost for miss-classifying Negative as Positive).

Looking at predicted probabilities generated by the model, I can see that it ranges from 0 to 1 for each class. i.e, I have instances where the predicted class (C1) has a probability lower than 0.5 ( for example: predicted class=C1 and predicted probability=0.1 ) - This is where the question arises: if P(C1) < 50% why is it classified as C1 (there are only two classes, C1 and C2)

Based on my understanding of decision trees, the predicted probabilities are often generated based on the percentage of test cases on the leaf node that were labeled in each class divided by the total number of instances hitting that leaf node. This method will dictate the probability for two classes must sum up to 1.

My question is why does the model classify an instance in class C1 if it has only got 0.1 confidence in it. Does a predicted probability of 0.1 on class C1 mean that there is a 0.9 confidence in it belonging to class C2? If so, why does it classify an instance in class C1 is it has less than 0.5 confidence in it?

My own theory is that this might be due to boosting and miss-classification cost and the way they are influencing the predicted class.

$\endgroup$
  • $\begingroup$ Probability theory is built on set of axioms: one of them is that sum of probabilities must sum to 1. Probability Axioms $\endgroup$ – Vitaly Portnoy Apr 27 '17 at 19:54
  • 2
    $\begingroup$ This axiom motivates the question itself - it does not provide an answer. $\endgroup$ – Greenish Apr 27 '17 at 20:10
  • $\begingroup$ I don't know anything about C5 but a model can associate confidences with its predictions apart from their probabilities, to describe how certain the prediction is. en.wikipedia.org/wiki/Uncertainty_quantification $\endgroup$ – Emre Apr 27 '17 at 20:43
  • $\begingroup$ In mathematics or logic, an axiom is an unprovable rule or first principle accepted as true because it is self-evident or particularly useful $\endgroup$ – Vitaly Portnoy Apr 27 '17 at 21:04
2
$\begingroup$

You are absolutely right to be skeptical about the results you are getting. I've been using boosted trees (not necessarily C5.0) for over 5 years, and I've never encountered results like what you've described here. Excluding the possibility that you simply had a typo in your code, I can think of two reasons:

  1. Low decision threshold: As is often the case in class imbalance problems (e.g., fraud detection, marketing response, bankruptcy, etc) a low probability could still indicate the minority class if the decision threshold is also low. In your case, if the model might have automatically decided to pick a threshold of 0.05, then 0.1 would be flagged as C1.

  2. Cost function: This is what you mentioned -- if your model has a high cost associated with missing true positives then 0.1 could be classified as C1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.