1
$\begingroup$

I would like to understand the output probabilities of a xgboost classifier (or any other decision tree ensemble based classifier) in the case of a multiclass problem.

For example: We have 5 different classes and a trained model on some data belonging to those classes. I would expect, given some test data, the model would give output probabilities of up to 100% when it is really sure the data belongs to some class. Obviously the probabilites still have to add up to 1.

On the contrary I was told that in the case of 5 different classes, the model would be already sure when outputting 20% (100/5) for the data. I also see this in a problem I am facing, the probabilities are never higher than 30%.

I would like to understand this apparent fact, ideally pointing me to a source where this is explained.

$\endgroup$

1 Answer 1

3
$\begingroup$

What you were told is a worst case scenario. With 5 labels, 20.01% is the lowest possible value that a model would need to choose one class over the other. If the probability for each of the 5 classes are almost equal then the probabilities for each would be approximately 20%. In this case, the model would be having trouble deciding which class is correct. About the 30% comment, while it seems low, keep in mind that the model mainly uses the probability to classify the most likely label, calculating a highly accurate probability value is not the first priority.

Here is a good reference. This paper discusses a few of the problems with probability accuracy and mentions some theories on how to improve the accuracy:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.381.4254&rep=rep1&type=pdf

It mentions (see excerpts below) how a probability estimation in classification is a complex and difficult problem, and that some models can be very accurate at classification but poor at probability estimation, and how multi-class models are much more difficult than a binary classification model.

Excerpt 1: "Probability estimation is known to be a quite hard problem, especially in comparison to standard classification. This comes at no surprise, noting that proper probability estimation is a sufficient but not necessary condition for proper classification: If the conditional class probabilities (1) are predicted accurately, an optimal classification can simply be made by picking the class with highest probability"

Excerpt 2: "More generally, the Bayes decision can be taken so as to minimize any loss in expectation. On the other hand, a correct classification can also be obtained based on less accurate probability estimates. In fact, the classification will remain correct as long as the estimated probability is highest for the true class. Or, stated differently, an estimation error will remain ineffective unless it changes the result of the arg max operation in (2). This is also the reason for why methods like naive Bayes show competitive performance in classification despite producing relatively inaccurate probability estimates [7]."

Excerpt 3: Methods like naive Bayes and decision trees are multi-class classifiers and can in principle be used to produce probability estimates in this setting. In practice, however, one often prefers to estimate probabilities in the two-class setting, especially because estimating a single probability (of the positive class) is much simpler than estimating K − 1 probabilities simultaneously

Example with probabilities close to 1:

from sklearn import datasets 
from sklearn.metrics import confusion_matrix 
from sklearn.model_selection import train_test_split 
iris = datasets.load_iris()   
X = iris.data 
y = iris.target  
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0) 
from sklearn.tree import DecisionTreeClassifier 
dtree_model = DecisionTreeClassifier(max_depth = 2).fit(X_train, y_train) 
dtree_predictions = dtree_model.predict(X_test) 

print(dtree_model.predict_proba(X_test) )

[[0. 0.02564103 0.97435897]
[0. 0.91666667 0.08333333]
[1. 0. 0. ]
[0. 0.02564103 0.97435897]
...

$\endgroup$
3
  • $\begingroup$ This seems intuitive and I'll accept your answer but can you point me in a direction where this could be cited from? $\endgroup$
    – N. Kiefer
    Jun 25, 2020 at 7:03
  • $\begingroup$ I added some citations above, along with more details on the issues with probability estimation $\endgroup$
    – Donald S
    Jun 25, 2020 at 15:02
  • 1
    $\begingroup$ FYI, I added an example with high probability values for a simple 3 class multiclass classification. There are values close to 1 in the output. So you can see your initial intuition was correct. $\endgroup$
    – Donald S
    Jun 26, 2020 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.