The blue dots are the raw data and the line is my logistic regression. The line is quite straight and not sigmoidal as I would have expected. I suspect there is something wrong in my gradient descent equation but I don't understand the maths well enough to find the mistake.

enter image description here

This is the code I used to generate the logistic regression:

import numpy as np
import pandas as pd
import matplotlib 
import matplotlib.pyplot as plt
from scipy import stats

def logit(z):
    return (1/(1+np.exp(-z)))

def cost(data, weights):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    cost = np.sum(y_val*scores - np.log(1 + np.exp(scores)))
    return cost

def gradient_descent(data, weights, learning_rate): 
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    weights -= (learning_rate/m)*np.dot(np.transpose(x_val),(logit(scores)-y_val))
    return weights 

dataset = pd.read_csv("/Users/An/Desktop/data/glass.csv") 
dataset.sort_values('Al', inplace=True) # sort by ascending "Al" values
dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})
ones = np.ones(len(dataset))
data = np.stack((ones,dataset["Al"], dataset["binary"]), axis=-1)

weights = np.random.rand(2)

def log_reg(data, weights):
    for i in range(9000):
        weights = gradient_descent(data, weights, 0.000001)
        loss = cost(data, weights)
    return weights

w = log_reg(data, weights)

if would suggest you implement the following formulas:

  • Sigmoid activation function

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

  • Output (prediction) formula

$$\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$$

  • Error function

$$\text{Error}(y, \hat{y}) = - y \log(\hat{y}) - (1-y) \log(1-\hat{y})$$

  • The function that updates the weights

$$ w_i \longrightarrow w_i + \alpha (y - \hat{y}) x_i$$

$$ b \longrightarrow b + \alpha (y - \hat{y})$$

If you want the corresponding Python code for the above just let me know. I also have it but I guess it would be more useful for you (from the point of view of learning) to implement it yourself. In any case, if you want it, I can update my answer with the Python code.

Source code:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def output_formula(features, weights, bias):
    return sigmoid(np.dot(features, weights) + bias)

def error_formula(y, output):
    return - y*np.log(output) - (1 - y) * np.log(1-output)

def update_weights(x, y, weights, bias, learnrate):
    output = output_formula(x, weights, bias)
    d_error = y - output
    weights += learnrate * d_error * x
    bias += learnrate * d_error
    return weights, bias
  • $\begingroup$ I just realised that my weights differ from the ones predicted by the scikit implementation by a factor of 2. Could you please upload your code so that I can see where I've gone wrong? $\endgroup$
    – Anya
    Jan 26 '19 at 10:28
  • $\begingroup$ Sure, I have just updated the answer. $\endgroup$
    – Alberto
    Jan 26 '19 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.