In which cases are non-linear learning methods preferred than logistic regression in classification problems?

Question

We know that neural networks and other learning methods can have better performance relative to logistic regression in some non-linear classification problems. But, it is known too that logistic regression can separate classes with a line that can be curvy if only we add more predictors that are a square, cube, etc of the given predictors (still considered a linear decision boundary).

So my question is, can we theoretically solve any classification problem through logistic regression, or are there limitations that I do not realize that make other non-linear learning methods mandatory to certain classification problems?

Answer 1

There are several assumptions that are needed to be satisfied in order for logistic regression to work. From personal experience the two most important assumptions:

1. Features need to be not (or a little) correlated with each other, because if they are correlated then the change of one feature indicates also the change of the other that's a problem when you try to solve the optimization problem
2. May some features not be ordinal

You can also check here for more information.

Other learning methods such as RandomForest require no such assumptions and may work better for some datasets. But keep in mind the no free lunch theorem, most of the times you need to get your hands dirty and test several learning models before you decide which works best for your data.