4
$\begingroup$

I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes probabilities?

Let's say our model is two trees. A terminal node of tree 1 has example 14 in a node with 20% class 1, 60% class 2, and 20% class 3. A terminal node of tree 2 has example 14 in a node with 100% class 2. Then our prediction for training example 14 is [10%, 80%, 10%].

Why use Softmax instead of this averaging approach?

Note: I am looking to apply this knowledge to understanding xgboost better, as well as a simple 1 tree classification model.

$\endgroup$

2 Answers 2

0
$\begingroup$

Depending on the parameters you used for your model, it may not be calibrated in probabilities. That is, your model output a score, that is helpfull to give a relative order between your instance, but the score may not reflect the real % chance of the output happening.

Softmax, will at least garanty that your output are between 0 and 1 and sum to one. This will give you an output that is nearer to a probability (but that may not be enough to be calibrated to 'historical' probabilities).

PS: I don't think there is a link with xgboost, unless you have very specific options.

$\endgroup$
0
$\begingroup$

To be clear, I think the softmax approach you mention is the generic one-vs-rest approach, in which a binary classifier is trained for each class, and their predicted probabilities are softmax'ed.

The averaging multiclass trees approach is used by (at least) some implementations, including scikit-learn's random forest.

Gradient boosting is rather different, in that the targets each tree is trained for are different, being the gradient of the loss function (maybe with hessian information or regularization). Nevertheless, I think it's possible, and perhaps catboost uses multioutput trees targeting each component of the gradients, while xgboost and lightgbm use one-vs-rest and softmax.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.