What makes binary cross entropy a better choice for binary classification than other loss functions?

Question

I'm reading this post where I came across this quote "Cross-entropy is the default loss function to use for binary classification problems."

But what about it makes it the default and presumably best loss function for binary classification?

Answer 1

As mentioned in the blog, cross entropy is used because it is equivalent to fitting the model using maximum likelihood estimation. This on the other hand can be interpreted as minimizing the dissimilarity between the empirical distribution of training data and the the distribution induced by the model (measured as Kullback-Leibler divergence, hence the name cross-entropy).

Now you may ask why maximizing the likelihood. Maximum Likelihood estimators have nice asymptotical properties (they are the best estimators in terms of convergence rates), they are consistent and statistically efficient.

Another advantage is that you don't need to think much about how to the define a cost function. The ML framework gives you the cost function right away as long as you have a model that specifies $P(y|x)$.

Also, directly optimizing the cross-entropy leads naturally to well-calibrated probabilities compared to, say, a random forest. That means you can interpret the resulting probabilities as you would expect, namely that of all the cases with about x% predicted probability, x% should belong to the positive class.

There is another important point. Given the prevalence of gradient-based learning algorithms (especially in deep learning), the logarithm in the cross-entropy will undo any exponential behavior that is often given through popular activation/output units like the sigmoid/softmax. Thus, the logarithm will avoid that the gradient saturates for extreme values. Large gradients are important for gradient-descent algorithms to make sufficient progress in each iteration. (There are more details, for example, in the "deep learning book" in chapter 6.2)