1
$\begingroup$

I am building logistic regression from scrap.

The simplified cost function I am using is (from machine learning course on coursera): enter image description here

in specific case during learning, one observation in training set y is 0 - but the specific choice of betas in:

enter image description here

makes g(z) = h(x) = 1 , because e.g. z > 50.

in this case my right side od J is (1 - 0) * log(1 - 1) what is -inf (I am doing my calculations in python)

I understand that in this case value of cost function should be high because the probability of y = 1 is very big while the truth is that it actually is 0.

Is the problem approximation of g(50) being 1 instead of something like: 0.999999? Or there is some more fundametal error with my logic?

because this specific example the summation of cost of all observations is nan (not a number) in my code.

$\endgroup$
1
$\begingroup$

In practice, an offset is used to avoid log explosion due to values close to zero. For example $\hat{\text{log}}(x)=\text{log}(x + \text{1e-6})$.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Ha. Didn't think about it. Thanks! $\endgroup$ – Mateusz Konopelski Mar 18 '19 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.