0
$\begingroup$

I am new to ML and I created a super basic logistic regression example with 4 points on the $x$ line that belong to two classes:

points = [[1, 1]]
points = points + [[2, 0]] 
points = points + [[1.5, 1]]
points = points + [[2.5, 0]]

data   = np.array([x[:-1] for x in points])
target = [int(x[-1]) for x in points]

alg = LogisticRegression()
alg.fit(data, target)
print (alg.coef_)
print (alg.intercept_)

print (alg.predict_proba(1))
print (alg.predict_proba(1.5))
print (alg.predict_proba(2))
print (alg.predict_proba(2.5))

I expected the model to understand that anything greater than 1.5 belongs to class 0, and anything smaller than 2 belongs to class 0. However, everything is predicted to be in class 0. Also, in this case , what is the meaning for the coef and intercept? More precisely, is there a way to deduce from the coef and intercept where the model thinks the points switch from class 0 to class 1?

$\endgroup$
1
$\begingroup$

What is the meaning for the coef and intercept

A (binary) logistic regression algorithm tries to determine whether the data $x$ belongs to class 0 or class 1 by the value $f(x)=\omega x+b$. If the value $f(x)>0$, the algorithm believe $x$ is more likely to be in class 1; while if $f(x)<0$, then $x$ is more likely to be in class 0.

In your code alg.coef_ is the $\omega$ above, and alg.intercept_ is the $b$ above.

Why everything is predicted to be in class 0

If you run your code, you should see the result alg.coef_=-0.354, alg.intercept_=0.307. Therefore your alg is calculating $f(x)=-0.354x+0.307$. Now if you plug in your data, i.e. $x=$1, 2, 1.5 and 2.5 respectively, you should get the value $f(x)=$-0.047,-0.401,-0.224 and -0.579 respectively.

As you see, all the four $f(x)$ values are smaller than 0, therefore your alg determines that they all belong be class 0.

But still why? Why the result is not what I expected?

Here is were regularization comes in. In the context of logistic regression, the learning algorithm regularizes that the learned parameter, $w$, should not be too large. To be specific, by default it regularizes the $l_2$ norm norm of $w$. In your case $w$ is a scalar, then $l_2$ norm is its absolute value.

In general, regularization is used to prevent overfitting. But that's another broad topic.

If you want your algorithm to behave as you expected (classify 1 and 1.5 to class 1, and classify 2 and 2.5 to class 0) , there are 2 ways.

  • You tell you program to use extremely weak regularization. That can be achieved by passing a large value to the C parameter in alg = LogisticRegression() (default value is 1.0, see detail), e.g.
alg = LogisticRegression(C=1000)

In this case your algorithm should return alg.coef_=-9.894, alg.intercept_=17.156, and it can classify your four data points correctly. Notice that the absotely value of alg.coef_ is now much larger than before.

  • Provide your algorithm with more data.

    For example, simply repeat your four data points 100 times:

data = [x[:-1] for x in points]*100 target = [int(x[-1]) for x in points]*100

Then you can get alg.coef_=-4.438, alg.intercept_=7.594 without setting C. In this case the same default regularization strength is still applied, but the additional data provide stronger evidence for your algorithm to believe that the alg.coef_ should be allowed larger (in absolutely value).

$\endgroup$
2
  • $\begingroup$ It does calculate sigmoid(f(x)). But inside your sigmoid function f(x)=ωx+b. The sigmoid function translates f(x) into a probability. $\endgroup$ – user12075 Sep 10 '18 at 0:38
  • $\begingroup$ Yes, I deleted my comment because I figured it out $\endgroup$ – user Sep 10 '18 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.