1
$\begingroup$

I'm quite used to seeing functions like log-loss, RMSE, cross entropy as objective functions and it's easy to imagine why minimizing these would give us the best model. What's difficult to imagine is how XGBoost uses softmax, a function used to normalize the logits, as a cost function. As mentioned in the docs here.

How can a softmax function be minimized?

$\endgroup$
2
$\begingroup$

It's not softmax to be minimized, but crossentropy loss function, which is based on softmax. Crossentropy is calculated on a softmax output, that's why they are a standard couple in ML. Tree-based classifiers find "cuts", or portions of the variables' space in a way that minimizes the entropy of a dataset.

If you want to explore the relationship between softmax and crossentropy further, you can start with the nice explanations provided here. If you want to dig deeper, you can find a very detailed and technical explanation here.

$\endgroup$
  • $\begingroup$ Okay!! So, is it the crossentrpy loss function that is minimzed even in the case of softprob? If yes, what difference does using softprob instead of softmax make exactly? Is it just while making the predictions that softprob gives the probability of each class and softmax gives the class with max probability or do they also differ in how the model actually learns? $\endgroup$ – p0712 Oct 15 at 12:33
  • 1
    $\begingroup$ Based on the crossentropy formula I'd say that softprob and softmax would return different results. I also think that if your model is a very good classifier, there shouldn't be serious differences between the two. However ML is more an art than a science, my suggestion is try both if you can and see what works best. $\endgroup$ – Leevo Oct 15 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.