Here is my logisticRegression class I developed to do gradient descent. There is this one line I marked as problematic

import numpy as np
class logisticRegression():
    """Logistic Regression classifier

    alpha : float
        Learning rate(between 0.0 and 1.0).
    iters : int
        Number of iterations.

    w_ : 1d-array
        Weights after fitting.
    def __init__(self, alpha = 0.001, iters = 100000):
        self.alpha = alpha
        self.iters = iters

    def fit(self, X, y):
        """Fit training data
        X : array-like, shape = [n_samples, n_features]
            Training vectors
        y : array-like, shape = [n_samples]
            Target values

        self : object"""
        # Initialize weight
        self.w_ = np.zeros(1 + X.shape[1])

        self.errors_ = []
        m = X.shape[0]
        x0 = np.ones(X.shape[0])

        for _ in range(self.iters):
            h = self.hyp(X)
            gradient = (X.T)@(h - y)/m
            self.w_[1:] -= self.alpha*gradient
            self.w_[0] -= self.alpha*x0@(h - y)/m # This line is problematic !!!
        return self

    def sigmoid(self, z):
        """Compute sigmoid"""
        return 1/(1+ np.exp(-z))

    def hyp(self, X):
        """Compute hypothesis (probability)"""
        return self.sigmoid(self.w_[0] + X@self.w_[1:] )

I got wrong result. But if I rewrite this line as:

self.w_[0] -= x0@(h - y)/m # Remove the learning rate term

Then I got correct result. But this doesn't seem right. Did I oversee something here?


For those who care, it turns out that removing the learning rate just makes the gradient happen faster. If alpha not removed, things still converge but incredibly slowly (it takes millions loops to converge) My theory is that my math was right, but plain gradient descent is computationally expensive.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.