11
$\begingroup$

For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data.

For example:

patient, Attr1, Attr2, Attr3, .., Label
002      90.0   8.0    98.0 ...   ? ===> predict drug A

How can I calculate the confidence or probability of the prediction result of drug A?

$\endgroup$
3

1 Answer 1

11
$\begingroup$

What data mining package do you use?

In sklearn, the DecisionTreeClassifier can give you probabilities, but you have to use things like max_depth in order to truncate the tree. The probabilities that it returns is $P=n_A/(n_A+n_B)$, that is, the number of observations of class A that have been "captured" by that leaf over the entire number of observations captured by that leaf (during training). But again, you must prune or truncate your decision tree, because otherwise the decision tree grows until $n=1$ in each leaf and so $P=1$.

That being said, I think you want to use something like a random forest. In a random forest, multiple decision trees are trained, by using different resamples of your data. In the end, probabilities can be calculated by the proportion of decision trees which vote for each class. This I think is a much more robust approach to estimate probabilities than using individual decision trees.

But random forests are not interpretable, so if interpertability is a requirement, use the decision tree like I mentioned. You can use grid search to maximize the ROC AUC score by changing hyperparameters such as maximum depth to find whatever decision tree gives the most reliable probabilities.

$\endgroup$
4
  • $\begingroup$ A link to random forest: scikit-learn.org/stable/modules/generated/… Or go to another implementation in the blog post $\endgroup$
    – borgr
    Jan 21, 2021 at 15:06
  • $\begingroup$ Very nice answer. Your blogpost link is broken though.. $\endgroup$
    – Rafs
    Jun 8, 2022 at 9:05
  • $\begingroup$ @RTD, thank you, my blog is no longer up. I have removed the link. $\endgroup$ Jun 8, 2022 at 13:14
  • $\begingroup$ A pity. I was really interested to see why a decision tree isn't recommended to estimate probabilities. I thought if we put the accuracy of the model in mind, and look at the probabilities, we can have a good representation of what the underlying probabilities are. We can say: with an accuracy of 70%, the positive class probability at this leaf is 0.8. Perhaps we can scale the probabilities by the accuracy - eccentric I know, just a rough idea. Not sure if you can elaborate on that, but thank you in any case. $\endgroup$
    – Rafs
    Jun 9, 2022 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.