We know that there are many different types of classification algorithms. But among the different categories of classification algorithms, which algorithms are suitable for binary classification and which are suitable for more classes, and why?
2 Answers
$\begingroup$
$\endgroup$
13
If you want to be highly literal, logistic regression is excellent for binary classes but completely inappropriate for $3+$ classes. No worries: there is multinomial logistic regression, the theory of which mimics binary logistic regression (one might consider logistic regression to be a special case of multinomial logistic regression). Depending on the sophistication of my audience, I might be comfortable referring to "logistic regression" and leaving it to them to realize that I mean "multinomial" logistic regression when there are $3+$ categories and "binary" logistic regression when there are $2$ categories.
Random forest can do the binary case but also the multiclass case. Ditto for k-nearest neighbors, support vector machines, and neural networks.
I cannot think of a model for binary classes that lacks a multiclass analogue.
-
-
1$\begingroup$ @AMZ What would constitute mathematical suitability? $\endgroup$– DaveNov 30, 2021 at 1:08
-
1$\begingroup$ I think SVMs can per se only do binary classification, since it works with a single separating hyperplane. If you want a multiclass SVM, you need to split the multiclass problem into multiple binary decisions (e.g. with one-vs-one or one-vs-all strategies). One could pursue the same approach with logistic regression (loosing inference statistics in the process). $\endgroup$– AEFNov 30, 2021 at 9:24
-
1$\begingroup$ @user2974951 yes, thats true, but the question and answer are about binary und multiclass problems and not about the difference between hard and soft classification. $\endgroup$– AEFNov 30, 2021 at 9:45
-
1$\begingroup$ There is a proper multiclass extension of the SVM (not just one-vs-one and one-vs-all reduction to multiple binary SVMs), the "Crammer-Singer multiclass SVM". $\endgroup$– isarandiNov 30, 2021 at 17:18
$\begingroup$
$\endgroup$
The answer is that there is no one answer. The choice of machine learning algorithms - whether for binary or multi-class classification - very much depends upon your data and your application:
- how big is your training set?
- how balanced is the training set?
- how many dependent variables does it contain?
- what is the separability of your two classes on those variables?
- what kind of accuracy/recall do you need?
- Is the result driving life or death decisions, or helping you decide what to eat for afternoon snack?
- Are you doing this once, or will you be developing multiple models?
For example, if you have a simple, linearly separable problem with a small training set, then a linear discriminant solution (Fisher) makes the most sense, but if you have a complex separability boundary with a small training data, then LR, SVMs or decision trees may be better, or if you have a large amount of data you can use neural nets. You should also consider how much time you want to invest in the model. Generally neural nets are a fairly big investment in time, but simpler models, such as LR and KNN can be implemented in Excel.
That said, more information on the data and the application might allow us to provide better suggestions.
