3
$\begingroup$

I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, but I just can't understand what is the importance of finding the roots or using second derivatives.

P.S. I know the idea of Newton Raphson, I am wondering why do we need to use this method and find the zero's or roots of function? What does it want to tell us for logistic regression for instance?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

This blogpost gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are either in a maximum, a minimum, or a saddle point (the three of them are critical points). When using the cross-entropy loss function in logistic regression, it can be proved that this loss function has only one critical point, and this critical point corresponds to the global minimum. For this reason, finding a root of the gradient is equivalent to finding the only critical point, therefore the global minimum of the cross-entropy loss function.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you very much for your response @David. Hmm, let's say we have a sample function that averages the set of data points. What is the advantage/purpose of finding the roots only on those encircled (red) intersections (image: i.imgur.com/DHR0Wa2.png) of the curve with respect to x axis? (to be continued) $\endgroup$
    – kindgnice
    Commented May 2, 2018 at 17:41
  • $\begingroup$ (continuation) If you're saying that we can find the "minimum, maximum, saddle point" in those encircled (red) points, what is the vital purpose of doing it? Does it have some correlation of finding least squares? Sorry, it's just not clear to me at all. I realized finding those roots are not beneficial compared to finding the least squares (which makes sense because at least the idea of least squares gives you the least square error). $\endgroup$
    – kindgnice
    Commented May 2, 2018 at 17:44
  • $\begingroup$ I just don't find the importance of finding these roots of the gradient (well, as far as I know, gradient is somewhat the "runs" of the slope). $\endgroup$
    – kindgnice
    Commented May 2, 2018 at 17:47
  • $\begingroup$ This image looks intuitive for me from your link (image:thelaziestprogrammer.com/assets/images/plotly/…), however, it tries to find the purple which is the "max" of this function. So this means, we don't necessarily need to assume "Newton's method" to be always located/passing through the x axis(like the typical root finding in polynomials)? $\endgroup$
    – kindgnice
    Commented May 2, 2018 at 17:55
  • 1
    $\begingroup$ You really have to stop thinking about finding the root of the function, the Newton method in here is used to find a root of the gradient of the function. And, yes, using logistic regression cost function, finding a root of the gradient is the same as finding the global minimum. $\endgroup$
    – David Masip
    Commented May 3, 2018 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.