# Why is logistic regression not sigmoidal?

The blue dots are the raw data and the line is my logistic regression. The line is quite straight and not sigmoidal as I would have expected. I suspect there is something wrong in my gradient descent equation but I don't understand the maths well enough to find the mistake.

This is the code I used to generate the logistic regression:

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
from scipy import stats

def logit(z):
return (1/(1+np.exp(-z)))

def cost(data, weights):
x_val = data[:,0:2]
y_val = data[:,2]
m = len(x_val)
scores = np.dot(x_val, weights)
cost = np.sum(y_val*scores - np.log(1 + np.exp(scores)))
return cost

x_val = data[:,0:2]
y_val = data[:,2]
m = len(x_val)
scores = np.dot(x_val, weights)
weights -= (learning_rate/m)*np.dot(np.transpose(x_val),(logit(scores)-y_val))
return weights

dataset.sort_values('Al', inplace=True) # sort by ascending "Al" values
dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})
ones = np.ones(len(dataset))
data = np.stack((ones,dataset["Al"], dataset["binary"]), axis=-1)

weights = np.random.rand(2)

def log_reg(data, weights):
for i in range(9000):
loss = cost(data, weights)
return weights

w = log_reg(data, weights)


if would suggest you implement the following formulas:

• Sigmoid activation function

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

• Output (prediction) formula

$$\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$$

• Error function

$$\text{Error}(y, \hat{y}) = - y \log(\hat{y}) - (1-y) \log(1-\hat{y})$$

• The function that updates the weights

$$w_i \longrightarrow w_i + \alpha (y - \hat{y}) x_i$$

$$b \longrightarrow b + \alpha (y - \hat{y})$$

If you want the corresponding Python code for the above just let me know. I also have it but I guess it would be more useful for you (from the point of view of learning) to implement it yourself. In any case, if you want it, I can update my answer with the Python code.

Source code:

def sigmoid(x):
return 1 / (1 + np.exp(-x))

def output_formula(features, weights, bias):
return sigmoid(np.dot(features, weights) + bias)

def error_formula(y, output):
return - y*np.log(output) - (1 - y) * np.log(1-output)

def update_weights(x, y, weights, bias, learnrate):
output = output_formula(x, weights, bias)
d_error = y - output
weights += learnrate * d_error * x
bias += learnrate * d_error
return weights, bias

• I just realised that my weights differ from the ones predicted by the scikit implementation by a factor of 2. Could you please upload your code so that I can see where I've gone wrong? – Anya Jan 26 '19 at 10:28
• Sure, I have just updated the answer. – Alberto Jan 26 '19 at 19:16