When classes are non-linearly separable, which of the following methods performs better? Choose correct one :-

  1. Logistic Regression
  2. Random Forest
  3. K Nearest Neighbor Classification
  4. Linear Regression over most important features

Trail : I think Random Forest and K Nearest Neighbor Classification are the only non-linear classifiers here. But which one performed better? how to judge here?

  • 3
    $\begingroup$ Welcome to DataScienSE. Is this a homework question? If yes I think it's a really bad question, because there's no classifier which is better than any other for any dataset (independently from the question of linearly separable). $\endgroup$ – Erwan Jan 10 at 18:06
  • $\begingroup$ @Erwan Yes. It's one of the assignment. Also, no data are given. Since I'm learning the classifiers how to judge is very challenging for me. Also, thanks for the welcome. :) $\endgroup$ – Argha Jan 10 at 18:12
  • $\begingroup$ I agree with @Erwan's comment. Regarding the decision surface the model algorithm might create, I would say k-nearest neighbours could be more "free" in its shape, so maybe it is the intended answer to the question $\endgroup$ – German C M Jan 10 at 18:14
  • $\begingroup$ all methods listed have non-linear areas where they can perform good enough (including linear regression since it is based on most-important features which is a non-linear operation). . However the "better" qualification is misleading as there is no theoretical reason random forests are better than k-NN under all cases $\endgroup$ – Nikos M. Jan 10 at 19:10

It is unusual to have such a bad question as an assignement. Maybe it's just misguided phrasing by the question author, but I'm pretty sure that the vast majority of experts in ML consider the idea of "best ML algorithm in general" as a mistake.

First let me explain why this is a bad question: in ML there is no way to guarantee that a particular algorithm/method performs better than any other on any dataset in general. There can be exceptions under very specific constraints, but there is no algorithm which is universally the best for non-linearly separable classes. There is actually a theoretical result called the No Free Lunch Theorem which is often interpreted as a proof that there cannot be a universally best classification/regression algorithm.

Now that the context of this bad question is clearly established, as a subjective personal opinion my answer would be based on interpreting the question as: which algorithm is the least sensitive to linear separability? In this context I would say kNN because this method is not concerned at all about linear separability: a new instance is classified based on its closest instances in the training data, so it does not rely on separating the data points into groups at all.


Not the answer you're looking for? Browse other questions tagged or ask your own question.