I am learning machine learning and after reading through materials on logistic regression i attempted to implement logistic regression with gradient descent in python from scratch.

It works well for some cases but for some cases it results in mathematical error, which is understandable if we see the case below.

the cost function in logistic regression is -( ylog(predicted) + (1-y)log(1-predicted))

what happens when predicted is 1? code fails because it attempts to calculate log(1-1) = log(0) which is undefined. Explicitly we get this error in python

ValueError('math domain error')

Please help me in understanding how can i prevent this case.

Code is given below:

from numpy.random import RandomState

import pandas as panda
import matplotlib.pyplot as plot 
import random
from math import sqrt, exp, log
remote_location = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

def standard_deviation(values):
    average = sum(values) / len(values)

    variance = sum([(average - i)**2/len(values) for i in values])

    return sqrt(variance)

class LogisticRegression(object):

    def __init__(self, epochs, learning_rate, _x_training_set, _y_training_set, standardize = False, random_state = None):
        self.epochs = epochs
        self.learning_rate = learning_rate
        self.standardize = standardize        
        self._x_training_set = _x_training_set
        self._y_training_set = _y_training_set
        self.number_of_training_set = len(self._y_training_set)
        self.weights = []      
        self.random_state = RandomState(random_state if random_state else 1)

    def standardizeInputData(self):

        Standardizing of feature set means substracting the mean of
        each training sample from the feature value and dividing it by
        the standard deviation

        1. take average of j features from i th training sample . say avg
        2. calculate the variance of each j feature
        3. variance(j) = (avg - x(j))**2/len(features)
        4. standard deviation of x(j) = sq rt(variance(j))

        so standardized(x(j)) = x(j) - avg / standard deviation(x(j))

        temp = []

        for i in range(len(self._x_training_set)):

            mean = sum(self._x_training_set[i])/ len(self._x_training_set[i])
            std_deviation = standard_deviation(self._x_training_set[i])
            temp.append([ (j - mean)/std_deviation for j in self._x_training_set[i]])            

        return temp

    def setup(self):

        if self.standardize:
            self._x_training_set = self.standardizeInputData()

        self.initialize_weights(len(self._x_training_set[0]) + 1)

    def initialize_weights(self, number_of_weights):

        self.weights = list(self.random_state.normal(loc = 0.0, scale = 0.01, size = len(self._x_training_set[0]) + 1))

    def learn(self):

        epoch_data = {}
        error = 0

        for epoch in range(self.epochs):

            cost =0 

            for i in range(self.number_of_training_set):
                _x = self._x_training_set[i]
                _desired = self._y_training_set[i]
                _weight = self.weights

                weighted_sum = _weight[0] + sum([_weight[j+1] * _x[j] for j in range(len(_x))])

                guess = 1 / ( 1 + exp(- weighted_sum))

                error = _desired - guess 

                ## i am going to reset all the weights
                if error!= 0 :

                    ## resetting the bias unit
                    self.weights[0] = error * self.learning_rate
                    self.weights[1:] =[self.weights[j+1] + error * self.learning_rate * _x[j] \
                                            for j in range(len(_x))]

                    ## cost entropy loss function
                    cost+= - ( _desired * log(guess) + (1 - _desired) *log(1-guess))

            #saving error at the end of the training set        
            epoch_data[epoch] = cost ##summation of all such y predictions for a training set


    def predict(self, _x_test_data):

            Given algorithm has been trained using the #learn method
            this method will predict the y values based on the last
            values calculated for weights. This is because
            by the end of the learn method, algorithm has already
            converged as close to 0 error as it can
        prediction = []

        for i in range(len(_x_test_data)):

            weighted_sum = self.weights[0] +  \
                    sum([self.weights[j+1] * _x_test_data[i][j] \
                        for j in range(len(_x_test_data[i]))])

            guess = 1 / ( 1 + exp(- weighted_sum))

            prediction.append( 1 if guess >= 0.5 else 0)

        return prediction

client code:

import pandas as panda

from sklearn.model_selection import train_test_split
from predicting_logistic_regression import LogisticRegression
from sklearn.metrics import accuracy_score, mean_absolute_error
from sklearn import datasets

remote_location = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'

# data = panda.read_csv(remote_location)       
# _x_training_set = list(data.iloc[0:, [0,2]].values)
# _y_training_set = [0 if i.lower()!='iris-setosa' else 1 for i in data.iloc[0:, 4].values]

data = datasets.load_iris()
_x_training_set = data.data[:,[2,3]]
_y_training_set = data.target 

_x_train, _x_test, _y_train, _y_test = train_test_split( \
                                        _y_training_set, \
                                        test_size = 0.3, \
                                        random_state = 1, \
                                        stratify = _y_training_set)

random_generator_start = -1
random_generator_end = 1

logistic_regression = LogisticRegression( \
                learning_rate = 0.01, \
                epochs = 40, \
                _x_training_set = _x_train, \
                _y_training_set = _y_train,
                standardize= False

_y_predicted = logistic_regression.predict(_x_test)

print(accuracy_score(_y_test, _y_predicted))
print(mean_absolute_error(_y_test, _y_predicted))

While in theory you would never get values exactly equal to one or zero, in practice that's something that can indeed happen due to floating point arithmetic (if your values become too close to zero or one). You can prevent it by setting a minimum and maximum value for your 'guess' variable, and you are less likely to get into that situation if you add regularization. It's also less likely to happen in "harder" datasets in which you don't get nearly 100% accuracy - I'm assuming you are probably getting this error in some toy dataset like the iris one.

  • $\begingroup$ thank you for your response. i agree it is happening for toy data sets such as iris. howevever if you really look at the code, there is a potential issue. i use guess = 1/(1+exp(weighted_sum)) . now if weighted_sum is beyond 710, it is too huge a number leading to overflow issue.similarly there could be underflow issue as well. hence i handled the issue using normalization techniques. you can check my answer below. this comment is not allowing me to add code snippets $\endgroup$ – user1906450 Oct 15 '18 at 6:05

thnx to all the google search and multiple articles related to logistic regression issues, this is what i came up with.

if you look at the code there is a potential issue in this particular line:

weighted_sum = _weight[0] + sum([_weight[j+1] * _x[j] for j in range(len(_x))])
guess = 1 / ( 1 + exp(weighted_sum))

in cases where weighted_sum is larger than 710, the corresponding exp function gives such large values that it leads to overflow errors. similarly for real low numbers it can also lead to underflow issues.

in order tofix that, i have used normalization techniques. courtesy - https://stats.stackexchange.com/questions/70801/how-to-normalize-data-to-0-1-range

this is the updated code:

weighted_sum = [_weight[0]] + [_weight[j+1] * _x[j] for j in range(len(_x))]

normalized_weighted_sum =  (sum(weighted_sum) - min(weighted_sum))/ (max(weighted_sum) - min(weighted_sum))

guess = 1 / ( 1 + exp(normalized_weighted_sum))

this worked like a charm.


You could try clipping the values of 0 and 1 and use very close values to them instead in order to avoid NaN instances, for example you could tweak lines of codes like this one:

some variable= np.clip("some fuction",1e-15,1-1e-15) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.