3
$\begingroup$

Logistic regression is supposed to work well only on data that is linearly separable. As we can see in the pair plot, the data points heavily overlap. The logistic regression model is in fact showing better accuracy than Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data). Even the cross validation score is higher for my logistic regression model.

Pair Plot

$\endgroup$
11
$\begingroup$

Decision Tree, KNN, & Random Forest (Methods that are suitable for overlapping data)

This statement is false. All those methods are good when the decision surface (separating surface) has a highly nonlinear form. They act as a non-parametric local approximation - all parameters are not in fact parameters of the decision function but are meta parameters of the model. When the decision separator is linear then it is to be expected logistic regression to perform better.

To see if your data is linearly separable it is not enough to see pairwise scatter plots. Imagine to clouds of points, one nested inside another. If they are round, no matter how you project them in any pairwise axes they will look overlapped.

$\endgroup$
6
$\begingroup$

Your data is multidimensional, it is possible that any two dimensional projection overlaps while still existing an hyperplane on the original dimensionality that separates the two classes well

Say for instance you have 3 data points from 2 labels in 2d that are linearly separable X:(0,-1) O:(1,2) X:(4,3)

           X
     O

   X

In the x axis they look like. X. O. X In the y axis they look like. X. O. X They are not 1d-separable in any axis

Yet there's a line in 2d that separates them (y=x)

This gets easier to happen the higher you get(in dimensional space xD)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.