Can I turn any binary classification algorithms into multiclass algorithms using softmax and cross-entropy loss?

Question

Softmax + cross-entropy loss for multiclass classification is used in ML algorithms such as softmax regression and (last layer of) neural networks. I wonder if this method could turn any binary classification algorithms into a multiclass one? For instance,

If I am using a polynomial function for binary classification, the decision step being 'predict positive if the output of the polynomial is greater than 0, otherwise predict negative', then I could use k of such polynomials$\dagger$ for k-class classification, each polynomial $f_i(X)$ has its own set of parameters to be learned; the objective is then to minimize the KL divergence between the sample distribution of the one-hot label and $e^{f_i(X)}/\sum_{i=1}^{k} e^{f_i(X)}$ or equivalently, cross-entropy.

Now if $f(X)$ is linear, this is exactly softmax regression algorithm, would it work if $f(X)$ is a polynomial or factorization machine or any classification algorithms that output a real number?

Some disadvantages of this approach I have in mind:

1. the parameters scale linearly with the number of classes

2. the loss function may be non-convex and hard to optimize

3. the theoretical properties/guarantees of the original binary classifiers may be lost

And how does this compare to classical 1-v-1 or 1-v-all approaches?

$\dagger$ Only $k-1$ is needed since softmax is over-parameterized

Answer 1

Yes, it is possible to use softmax and cross-entropy loss to turn a binary classification algorithm into a multiclass classification algorithm. In general, this can be done by using multiple binary classifiers, each trained to differentiate between one of the classes and all other classes. The outputs of these binary classifiers can then be combined using the softmax function and the cross-entropy loss can be used to train the model to predict the correct class.

This approach has several disadvantages, as you mentioned. The number of parameters in the model scales linearly with the number of classes, which can make it difficult to train the model effectively with a large number of classes. Additionally, the loss function may be non-convex and difficult to optimize, which can make it challenging to find a good set of model parameters. Finally, the theoretical properties and guarantees of the original binary classifier may be lost when using this approach, which can impact the performance of the model.

In comparison to the 1-vs-1 and 1-vs-all approaches, the approach you described has the advantage of using the softmax function to normalize the outputs of the binary classifiers and allow for probabilistic predictions. This can make it easier to interpret the model's predictions and make them more meaningful in certain contexts. However, the 1-vs-1 and 1-vs-all approaches have the advantage of being more computationally efficient, as they require training fewer binary classifiers and have fewer model parameters in total. Additionally, these approaches can take advantage of the theoretical properties and guarantees of the original binary classifier, which can improve the performance of the multiclass model.