Logistic regression cost function is cross-entropy. It is defined as below:


This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such as : ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’.

Is it possible to analytically find the minimum? if yes, what can we say of the computing complexity?

  • 1
    $\begingroup$ Have you checked this post? It might be helpful. $\endgroup$ – Shadi Nov 1 '17 at 17:27

If you use just one neuron (linear->sigmoid), you can find the minimum with too much small error or maybe reach to the minimum using approaches like gradient descent or other optimization algorithms. The reason for finding the minimum is that cross entropy is a non-convex shape, but if you use sigmoid function as the activation function of logistic regression, the cross entropy cost function becomes convex and it is easy to find the only global minimum.

| improve this answer | |
  • $\begingroup$ I am searching for an analytical solution expression with a closed form. Looking at Shadi answer, it seems that the solution can't be easily expressed for most of logistic regression problems (more than two input variables...). $\endgroup$ – Theudbald Nov 26 '17 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.