Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm quite used to seeing functions like log-loss, RMSE, cross entropy as objective functions and it's easy to imagine why minimizing these would give us the best model. What's difficult to imagine is how XGBoost uses softmax, a function used to normalize the logits, as a cost function. As mentioned in the docs here.
How can a softmax function be minimized?
It's not softmax to be minimized, but crossentropy loss function, which is based on softmax. Crossentropy is calculated on a softmax output, that's why they are a standard couple in ML. Tree-based classifiers find "cuts", or portions of the variables' space in a way that minimizes the entropy of a dataset.
If you want to explore the relationship between softmax and crossentropy further, you can start with the nice explanations provided here. If you want to dig deeper, you can find a very detailed and technical explanation here.
