Simple logistic regression wrong predictions

Question

In the following example, the prediction obtained for $x=[0,0]$ after fitting the model is $1$, but expected is $0$.

X = np.array([
    [0, 0],
    [1, 0],
    [0, 1]
])

Y = np.array([0,1,1])

model = LogisticRegression()
model.fit(X,Y)

x = [0,0]
y = model.predict([x])[0]

It only works if I duplicate the training row of ($x=[0,0], y=[0]$)

But why?

Answer 1

Because you have a class imbalance and very little data. Your model is essentially working off of a prior probability that disproprtionately favors the positive class, and you didn't give the model enough data to escape this prior in the learning process. Consider the following detailed output from your model:

model.score(X,Y) # 0.6666
model.coef_      # array([[ 0.36566712,  0.36566712]])
model.intercept_ # array([ 0.18517658])

model.predict_proba(X)
# array([[ 0.45383769,  0.54616231],
#        [ 0.36566869,  0.63433131],
#        [ 0.36566869,  0.63433131]])

Your model correctly identifies that [0,0] is more likely to be negative than either of the other observations, but the model isn't able to escape its bias towards the positive class. If you duplicate your dataset a couple of times, the model will eventually have enough "evidence" to escape this bias. Geometrically, two things will happen when we give the model more data:

1. The discrimination boundary will shift towards the negative class. This will manifest as an increasingly negative intercept term.
2. The slope of the logistic curve will steepen, having the effect that a unit change in the inputs will have a larger impact on the outcome. If we give the model enough perfectly separable data, the curve will approach a step function and the model will output probabilities of 0 and 1. This manifests in the magnitude of the coefficients. With just three observations, the curve is fairly flat and changing the values of the inputs doesn't have much effect on the outcome probability, so the resulting class assignment is essentially determined by the intercept alone.

Contrast the results from your original model with the what happens when we replicate your dataset 100 times:

model2 = LogisticRegression()
X2 = np.matlib.repmat(X,100,1)
Y2 = np.matlib.repmat(Y,100,1).ravel()
model2.fit(X2, Y2)

model2.score(X,Y) # 1.0
model2.coef_      # array([[ 4.95489068,  4.95489068]])
model2.intercept_ # array([-2.00091433])
model2.predict_proba(X)

# array([[ 0.88089304,  0.11910696],
#        [ 0.04954891,  0.95045109],
#        [ 0.04954891,  0.95045109]])

Answer 2

This is because you have an imbalanced dataset towards class 0. I have taken a look on the logistic regression coefficient you get. On the below chart 1, I have plotted the decision boundary you get with your logistic regression. On the second chart, I have plotted what you have expected.

So why this difference and why does your logistic regression yield chart 1? The reason is even if it is a machine learning algorithm, it is not really smart. The logistic regression algorithm wants to minimize its cost fucntion (cross-entropy). Cross-entropy can be defined in a really simple way as the distance between your points and the decision boundary.

But, in your training case, you have two class-1 observations for only one class_0 observations. Thus the machine learning algorithm will see that in order to reach the lowest cost function as possible, the best choice is to promote the two class-1 observations than the only one class-0 observation. The cost function value is larger on chart 2 than on chart 1, this is why it has yielded chart 1 result. Majority wins !

To avoid this problem, here are three ideas you can do :

• Undersample / oversample your dataset in order to have a similar number of observations in each class of your training set.
• Change your observations class weight.
• Use an other cost function like Hinge loss that is the cost function used in SVM and where the goal is to maximize the margin between classes.

Answer 3

In the simplest language, your model "sees" more number of class 1 examples than class 0 examples (because of very little data), so the probability of having class 1 as output is higher. As others have pointed out, this is a class imbalance problem. So, even if you give [0,0] in prediction, it will return class 1 as output.

The current ratio of classes 1 and 0 in training set is 2:1. When you duplicate the data, ratio becomes 1:1, so the model now correctly predicts your example.