1
$\begingroup$

Linear Regression cost function:

$$J(\theta) = \frac{1}{2 m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$

where:

$$h_{\theta}(x) = \theta_0 + \theta_1 x_1$$

Logistic Regression cost function

$$J(\theta) = \frac{1}{m} \sum_{i=1}^m - (y^{(i)} \times log(h_{\theta}(x{(i)})) + (1-y{(i)}) \times log(1 - h_{\theta}(x{(i)})))$$

where:

$$h_{\theta}(x) = g(\theta_0 + \theta_1 x_1 + \theta_2 x_2)$$

Intuitively linear regression is easy to understand as it optimizes the average squared distance between hypothesis and training data. But in case of logistic regression, I failed to understand the cost function.

What does the logistic regression cost function represent?

$\endgroup$
  • 3
    $\begingroup$ It's negative log likelihood, so if you minimize this function, you consequently maximize likelihood of the data. They derive it for the case with 2 classes here. $\endgroup$ – nikkou Aug 26 '17 at 7:26
  • $\begingroup$ Likelihood function in this case is $\prod h_\theta (x^{(i)})^{y^{(i)}} (1 - h_\theta (x^{(i)})^{(1-y^{(i)})}$. Negative log likelihood is equal to your cost function (except for the multiplier). $\endgroup$ – nikkou Aug 26 '17 at 7:33
  • $\begingroup$ Atinesh: In this formulation $y^{(i)} \in \{0,1\}$, so either way one of the terms gets eliminated. It's a succint way of expression of defining this piecewise function. When you take the logarithm the products separate into sums. $\endgroup$ – Emre Aug 26 '17 at 8:16
  • $\begingroup$ I acknowledge it is not that compliant to post a link as an answer, but for the case in point the following video by Andrew Ng which can really help original poster youtube.com/watch?v=HIQlmHxI6-0 $\endgroup$ – tagoma Aug 27 '17 at 7:45
1
$\begingroup$

The way my intuition works for logistic regression is simple, say you are trying to classify if you have a picture with a dog on it, if that is the case, you output 1 (true), if not, you output 0 (false)

The cost function essentially represents the diff between what your model output (imagine, you have a dog in a picture, and your model outputs 0.75 instead of 1, your cost is 0.25).

Now, this is not a very mathematical way to put it, but at least it has helped me a lot to understand logistic regression.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.