1
$\begingroup$

Logistic regression cost function is cross-entropy. It is defined as below:

J(θ)=1m∑i=1m−(y(i)×log(hθ(x(i)))+(1−y(i))×log(1−hθ(x(i))))

This is a convex function. To reach the minimum, scikit-learn provides multiple types of solvers such as : ‘liblinear’ library, ‘newton-cg’, ‘sag’ and ‘lbfgs’.

Is it possible to analytically find the minimum? if yes, what can we say of the computing complexity?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Have you checked this post? It might be helpful. $\endgroup$
    – Shadi
    Commented Nov 1, 2017 at 17:27

1 Answer 1

Reset to default
0
$\begingroup$

If you use just one neuron (linear->sigmoid), you can find the minimum with too much small error or maybe reach to the minimum using approaches like gradient descent or other optimization algorithms. The reason for finding the minimum is that cross entropy is a non-convex shape, but if you use sigmoid function as the activation function of logistic regression, the cross entropy cost function becomes convex and it is easy to find the only global minimum.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am searching for an analytical solution expression with a closed form. Looking at Shadi answer, it seems that the solution can't be easily expressed for most of logistic regression problems (more than two input variables...). $\endgroup$
    – Theudbald
    Commented Nov 26, 2017 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.