# Super basic logistic regression example

I am new to ML and I created a super basic logistic regression example with 4 points on the $x$ line that belong to two classes:

points = [[1, 1]]
points = points + [[2, 0]]
points = points + [[1.5, 1]]
points = points + [[2.5, 0]]

data   = np.array([x[:-1] for x in points])
target = [int(x[-1]) for x in points]

alg = LogisticRegression()
alg.fit(data, target)
print (alg.coef_)
print (alg.intercept_)

print (alg.predict_proba(1))
print (alg.predict_proba(1.5))
print (alg.predict_proba(2))
print (alg.predict_proba(2.5))


I expected the model to understand that anything greater than 1.5 belongs to class 0, and anything smaller than 2 belongs to class 0. However, everything is predicted to be in class 0. Also, in this case , what is the meaning for the coef and intercept? More precisely, is there a way to deduce from the coef and intercept where the model thinks the points switch from class 0 to class 1?

# What is the meaning for the coef and intercept

A (binary) logistic regression algorithm tries to determine whether the data $x$ belongs to class 0 or class 1 by the value $f(x)=\omega x+b$. If the value $f(x)>0$, the algorithm believe $x$ is more likely to be in class 1; while if $f(x)<0$, then $x$ is more likely to be in class 0.

In your code alg.coef_ is the $\omega$ above, and alg.intercept_ is the $b$ above.

# Why everything is predicted to be in class 0

If you run your code, you should see the result alg.coef_=-0.354, alg.intercept_=0.307. Therefore your alg is calculating $f(x)=-0.354x+0.307$. Now if you plug in your data, i.e. $x=$1, 2, 1.5 and 2.5 respectively, you should get the value $f(x)=$-0.047,-0.401,-0.224 and -0.579 respectively.

As you see, all the four $f(x)$ values are smaller than 0, therefore your alg determines that they all belong be class 0.

# But still why? Why the result is not what I expected?

Here is were regularization comes in. In the context of logistic regression, the learning algorithm regularizes that the learned parameter, $w$, should not be too large. To be specific, by default it regularizes the $l_2$ norm norm of $w$. In your case $w$ is a scalar, then $l_2$ norm is its absolute value.

In general, regularization is used to prevent overfitting. But that's another broad topic.

If you want your algorithm to behave as you expected (classify 1 and 1.5 to class 1, and classify 2 and 2.5 to class 0) , there are 2 ways.

• You tell you program to use extremely weak regularization. That can be achieved by passing a large value to the C parameter in alg = LogisticRegression() (default value is 1.0, see detail), e.g.
alg = LogisticRegression(C=1000)

In this case your algorithm should return alg.coef_=-9.894, alg.intercept_=17.156, and it can classify your four data points correctly. Notice that the absotely value of alg.coef_ is now much larger than before.

• Provide your algorithm with more data.

For example, simply repeat your four data points 100 times:

data = [x[:-1] for x in points]*100 target = [int(x[-1]) for x in points]*100 

Then you can get alg.coef_=-4.438, alg.intercept_=7.594 without setting C. In this case the same default regularization strength is still applied, but the additional data provide stronger evidence for your algorithm to believe that the alg.coef_ should be allowed larger (in absolutely value).

• It does calculate sigmoid(f(x)). But inside your sigmoid function f(x)=ωx+b. The sigmoid function translates f(x) into a probability. Sep 10 '18 at 0:38
• Yes, I deleted my comment because I figured it out
– user
Sep 10 '18 at 0:44