# How do we solve this logistic regression question?

Can an exact answer for this question be found? By intuition, I think the answer is 0. But can someone explain the steps on how to solve this question? ## 2 Answers

This particular problem has perfect separation, so your intuition can be made rigorous. Otherwise, as gunes said, logistic regression is not easy to optimize by hand.

Write $$y = \beta_0 + \beta_1 f_1 + \beta_2 f_2 + \beta_3 f_3$$ for the linear model on log-odds.

As a bit of motivation first, notice that evaluating at $$x_6$$ just returns the intercept $$\beta_0$$. If our model can obtain perfect accuracy, then from $$x_1$$ we obtain that $$\beta_0+\beta_1 < 0$$, and from $$x_3$$ that $$\beta_0+\beta_2<0$$, but from $$x_5$$ that $$\beta_0+\beta_1+\beta_2>0$$. This is impossible if $$\beta_0\geq0$$, and so we expect that $$y_6=\beta_0<0$$ and for the output to be 0.

To make things precise, we can find $$\beta$$ that satisfies the three inequalities above, but that furthermore make the log-odds approach $$\pm \infty$$ as appropriate. Let $$\beta_0=-3\alpha$$, $$\beta_1=\beta_2=2\alpha$$, $$\beta_3=0$$. Then for each of $$x_1, x_2, x_3$$ we have $$y=-\alpha$$, and for $$x_4, x_5$$ we have $$y=+\alpha$$. Letting $$\alpha\to\infty$$ forces these log-odds toward $$\pm\infty$$, so that the log-likelihood goes to infinity.

In other (shorter) words, we have perfect separation by the plane $$-3+2f_1+2f_2=0$$.

It can be found, assuming a proper learning rate, a suitable threshold, and binary cross-entropy cost, since it translates this into a convex problem, in which we have one global optimum. We don't have closed form solution for logistic regression, but through gradient descent we can get to this optimum arbitrarily close. I'd suggest running a logreg from scikit-learn or a familiar library.