1
$\begingroup$

I am a complete beginner in coding and machine learning, and I've been tasked with learning what's under the hood of logistic regression (so I have pieced together the python code below) but I've been asked to figure out how to add bias into this code. I'm completely stuck on at what point it would need to be added into, I think at the point I am defining the hypothesis function - but if anyone would be able to point me in the right direction to figure this out I would really appreciate it.

If it helps, this logistic regression is being used to classify if a tumour is benign of malignant with the wisconsin breast cancer dataset (https://www.kaggle.com/uciml/breast-cancer-wisconsin-data)

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)

X = data["diagnosis"].map(lambda x: float(x))

X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]
X = np.array(X)
X = min_max_scaler.fit_transform(X)
Y = data["diagnosis"].map(lambda x: float(x))
Y = np.array(Y)


def Sigmoid(z):
    if z < 0:
        return 1 - 1/(1 + math.exp(z))
    else:
        return 1/(1 + math.exp(-z))
def Hypothesis(theta, x):
    z = 0
    for i in range(len(theta)):
    z += x[i]*theta[i]
return Sigmoid(z)enter preformatted text here

def Cost_Function(X,Y,theta,m):
sumOfErrors = 0
for i in range(m):
    xi = X[i]
    hi = Hypothesis(theta,xi)
    error = Y[i] * math.log(hi if  hi >0 else 1)
    if Y[i] == 1:
        error = Y[i] * math.log(hi if  hi >0 else 1)
    elif Y[i] == 0:
        error = (1-Y[i]) * math.log(1-hi  if  1-hi >0 else 1)
    sumOfErrors += error

const = -1/m
J = const * sumOfErrors
print ('cost is: ', J ) 
return J

def Cost_Function_Derivative(X,Y,theta,j,m,alpha):
sumErrors = 0
for i in range(m):
    xi = X[i]
    xij = xi[j]
    hi = Hypothesis(theta,X[i])
    error = (hi - Y[i])*xij
    sumErrors += error
m = len(Y)
constant = float(alpha)/float(m)
J = constant * sumErrors
return J

def Gradient_Descent(X,Y,theta,m,alpha):
new_theta = []
constant = alpha/m
for j in range(len(theta)):
    CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)
    new_theta_value = theta[j] - CFDerivative
    new_theta.append(new_theta_value)
return new_theta

initial_theta = [0,1]  
alpha = 0.01
iterations = 1000
Logistic_Regression(X,Y,alpha,initial_theta,iterations)
$\endgroup$

1 Answer 1

5
$\begingroup$

If I understood correctly, by bias you mean the intercept term in your model, that is, $\alpha$ in the equation $$ p(y=1|x) = \frac{1}{1+e^{-(\alpha + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}} $$

If it is the case, you can easily incorporate intercept by adding a colum of ones into your X:

X = np.hstack([np.ones([X.shape[0],1]), X])

So the first colum of $X$ is always one. Then, your model will look like

$$ p(y=1|x) = \frac{1}{1+e^{-(\beta_0 x_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_k x_x)}} $$ where $\beta_0$ is your bias term and $x_0$ always equals $1$. Thus, your model would be equivalent to the one with explicit intercept.

$\endgroup$
5
  • $\begingroup$ What is default value for β0 (bias term) and also for other weights of features (β1, β2, etc...)? $\endgroup$
    – haneulkim
    Jan 31, 2021 at 13:03
  • 2
    $\begingroup$ @Ambleu if you are talking about starting values for gradient descent, then exact zeros or small random numbers around zero would make do. As an alternative, you may try to initialize the logistic regression from the linear regression line by making them tangent at the center of your data. May be, it will save you a few gradient descent steps. But in general, starting values don't matter much for logistic regression, because its loss function is convex and if there is an optimum, gradient descent is guaranteed to converge to it from anywhere. $\endgroup$
    – David Dale
    Jan 31, 2021 at 16:55
  • 2
    $\begingroup$ @Ambleu I got curious and created a notebook which demonstrates that initialization of logistic regression with coefficients from linear regression can speed up training. gist.github.com/avidale/a640f7a8e353d9efdd79385e277caef1 $\endgroup$
    – David Dale
    Jan 31, 2021 at 20:10
  • $\begingroup$ Absolutely amazing! the way you approach curious question is very motivating. Thanks you for your knowledge and motivation! One thing I want to make sure, why in your gist does it say "inverse logistic function is sometimes called logit" why sometimes? is there a time it isn't called a logit function? After doing some research I couldn't find when it is not called a logit function. $\endgroup$
    – haneulkim
    Feb 1, 2021 at 1:49
  • 1
    $\begingroup$ @Ambleu pleas never mind about this "sometimes", I don't know why I wrote it. $\endgroup$
    – David Dale
    Feb 1, 2021 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.