I am building logistic regression from scrap.
in specific case during learning, one observation in training set y is 0 - but the specific choice of betas in:
makes g(z) = h(x) = 1 , because e.g. z > 50.
in this case my right side od J is (1 - 0) * log(1 - 1) what is -inf (I am doing my calculations in python)
I understand that in this case value of cost function should be high because the probability of y = 1 is very big while the truth is that it actually is 0.
Is the problem approximation of g(50) being 1 instead of something like: 0.999999? Or there is some more fundametal error with my logic?
because this specific example the summation of cost of all observations is nan (not a number) in my code.