# Logistic Regression Cost Function: Gives mathematical error since its attempting to calculate log(0)

I am learning machine learning and after reading through materials on logistic regression i attempted to implement logistic regression with gradient descent in python from scratch.

It works well for some cases but for some cases it results in mathematical error, which is understandable if we see the case below.

the cost function in logistic regression is -( ylog(predicted) + (1-y)log(1-predicted))

what happens when predicted is 1? code fails because it attempts to calculate log(1-1) = log(0) which is undefined. Explicitly we get this error in python

ValueError('math domain error')

Code is given below:

from numpy.random import RandomState

import pandas as panda
import matplotlib.pyplot as plot
import random
from math import sqrt, exp, log
remote_location = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

def standard_deviation(values):
average = sum(values) / len(values)

variance = sum([(average - i)**2/len(values) for i in values])

return sqrt(variance)

class LogisticRegression(object):

def __init__(self, epochs, learning_rate, _x_training_set, _y_training_set, standardize = False, random_state = None):
self.epochs = epochs
self.learning_rate = learning_rate
self.standardize = standardize
self._x_training_set = _x_training_set
self._y_training_set = _y_training_set
self.number_of_training_set = len(self._y_training_set)
self.weights = []
self.random_state = RandomState(random_state if random_state else 1)

def standardizeInputData(self):
"""

Standardizing of feature set means substracting the mean of
each training sample from the feature value and dividing it by
the standard deviation

1. take average of j features from i th training sample . say avg
2. calculate the variance of each j feature
3. variance(j) = (avg - x(j))**2/len(features)
4. standard deviation of x(j) = sq rt(variance(j))

so standardized(x(j)) = x(j) - avg / standard deviation(x(j))

"""
temp = []

for i in range(len(self._x_training_set)):

mean = sum(self._x_training_set[i])/ len(self._x_training_set[i])
std_deviation = standard_deviation(self._x_training_set[i])
temp.append([ (j - mean)/std_deviation for j in self._x_training_set[i]])

return temp

def setup(self):

if self.standardize:
self._x_training_set = self.standardizeInputData()

self.initialize_weights(len(self._x_training_set[0]) + 1)

def initialize_weights(self, number_of_weights):

self.weights = list(self.random_state.normal(loc = 0.0, scale = 0.01, size = len(self._x_training_set[0]) + 1))

def learn(self):

self.setup()
epoch_data = {}
error = 0

for epoch in range(self.epochs):

cost =0

for i in range(self.number_of_training_set):
_x = self._x_training_set[i]
_desired = self._y_training_set[i]
_weight = self.weights

weighted_sum = _weight[0] + sum([_weight[j+1] * _x[j] for j in range(len(_x))])

guess = 1 / ( 1 + exp(- weighted_sum))

error = _desired - guess

## i am going to reset all the weights
if error!= 0 :

## resetting the bias unit
self.weights[0] = error * self.learning_rate
self.weights[1:] =[self.weights[j+1] + error * self.learning_rate * _x[j] \
for j in range(len(_x))]

## cost entropy loss function
cost+= - ( _desired * log(guess) + (1 - _desired) *log(1-guess))

#saving error at the end of the training set
epoch_data[epoch] = cost ##summation of all such y predictions for a training set

print(epoch_data)

def predict(self, _x_test_data):
"""

Given algorithm has been trained using the #learn method
this method will predict the y values based on the last
values calculated for weights. This is because
by the end of the learn method, algorithm has already
converged as close to 0 error as it can
"""
prediction = []

for i in range(len(_x_test_data)):

weighted_sum = self.weights[0] +  \
sum([self.weights[j+1] * _x_test_data[i][j] \
for j in range(len(_x_test_data[i]))])

guess = 1 / ( 1 + exp(- weighted_sum))

prediction.append( 1 if guess >= 0.5 else 0)

print(prediction)
return prediction


client code:

import pandas as panda

from sklearn.model_selection import train_test_split
from predicting_logistic_regression import LogisticRegression
from sklearn.metrics import accuracy_score, mean_absolute_error
from sklearn import datasets

remote_location = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

# _x_training_set = list(data.iloc[0:, [0,2]].values)
# _y_training_set = [0 if i.lower()!='iris-setosa' else 1 for i in data.iloc[0:, 4].values]

_x_training_set = data.data[:,[2,3]]
_y_training_set = data.target

_x_train, _x_test, _y_train, _y_test = train_test_split( \
_x_training_set,\
_y_training_set, \
test_size = 0.3, \
random_state = 1, \
stratify = _y_training_set)

random_generator_start = -1
random_generator_end = 1

logistic_regression = LogisticRegression( \
learning_rate = 0.01, \
epochs = 40, \
_x_training_set = _x_train, \
_y_training_set = _y_train,
standardize= False
)

logistic_regression.learn()
_y_predicted = logistic_regression.predict(_x_test)

print(_y_predicted)
print(_y_test)
print(accuracy_score(_y_test, _y_predicted))
print(mean_absolute_error(_y_test, _y_predicted))


While in theory you would never get values exactly equal to one or zero, in practice that's something that can indeed happen due to floating point arithmetic (if your values become too close to zero or one). You can prevent it by setting a minimum and maximum value for your 'guess' variable, and you are less likely to get into that situation if you add regularization. It's also less likely to happen in "harder" datasets in which you don't get nearly 100% accuracy - I'm assuming you are probably getting this error in some toy dataset like the iris one.

• thank you for your response. i agree it is happening for toy data sets such as iris. howevever if you really look at the code, there is a potential issue. i use guess = 1/(1+exp(weighted_sum)) . now if weighted_sum is beyond 710, it is too huge a number leading to overflow issue.similarly there could be underflow issue as well. hence i handled the issue using normalization techniques. you can check my answer below. this comment is not allowing me to add code snippets Oct 15 '18 at 6:05

thnx to all the google search and multiple articles related to logistic regression issues, this is what i came up with.

if you look at the code there is a potential issue in this particular line:

weighted_sum = _weight[0] + sum([_weight[j+1] * _x[j] for j in range(len(_x))])
guess = 1 / ( 1 + exp(weighted_sum))


in cases where weighted_sum is larger than 710, the corresponding exp function gives such large values that it leads to overflow errors. similarly for real low numbers it can also lead to underflow issues.

in order tofix that, i have used normalization techniques. courtesy - https://stats.stackexchange.com/questions/70801/how-to-normalize-data-to-0-1-range

this is the updated code:

weighted_sum = [_weight[0]] + [_weight[j+1] * _x[j] for j in range(len(_x))]

normalized_weighted_sum =  (sum(weighted_sum) - min(weighted_sum))/ (max(weighted_sum) - min(weighted_sum))

guess = 1 / ( 1 + exp(normalized_weighted_sum))


this worked like a charm.

You could try clipping the values of 0 and 1 and use very close values to them instead in order to avoid NaN instances, for example you could tweak lines of codes like this one:

some variable= np.clip("some fuction",1e-15,1-1e-15)