3
$\begingroup$

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes probabilities?

Let's say our model is two trees. A terminal node of tree 1 has example 14 in a node with 20% class 1, 60% class 2, and 20% class 3. A terminal node of tree 2 has example 14 in a node with 100% class 2. Then our prediction for training example 14 is [10%, 80%, 10%].

Why use Softmax instead of this averaging approach?

Note: I am looking to apply this knowledge to understanding xgboost better, as well as a simple 1 tree classification model.

$\endgroup$
0
$\begingroup$

Depending on the parameters you used for your model, it may not be calibrated in probabilities. That is, your model output a score, that is helpfull to give a relative order between your instance, but the score may not reflect the real % chance of the output happening.

Softmax, will at least garanty that your output are between 0 and 1 and sum to one. This will give you an output that is nearer to a probability (but that may not be enough to be calibrated to 'historical' probabilities).

PS: I don't think there is a link with xgboost, unless you have very specific options.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.