I am trying to manually implement the irls logistic regression (Chapter 4.3.3 in Bishop - Pattern Recognition And Machine Learning) in python.
For updating the weights, I am using $w' = w-(\Phi^TR\Phi)^{-1}\Phi^T(y-t)$
However I am not getting satisfying results, also my weights are growing unbounded in each iteration.

I've written this code so far:

def y(X, w):
    return sigmoid(X.dot(w))

def R(y):
    R = np.identity(y.size)
    R = R*(y*(1-y))
    return R

def irls(X, t):
    w = np.ones(X.shape[1])
    w = w.reshape(w.size, 1)
    t = np.array(list(map(lambda x: 1 if x else 0, t)))
    t = t.reshape(t.size, 1)

    #after 3 iterations matrix is singular
    for i in range(3):
        y_ = y(X,w)
        w = w - np.linalg.inv(X.T.dot(R(y_)).dot(X)).dot((X.T).dot(y_-t))
    return w

where X is my design matrix and t is my target vector comprising Boolean values (data is from https://archive.ics.uci.edu/ml/datasets/Mice+Protein+Expression).

Any help is appreciated pointing out where I went wrong.

  • $\begingroup$ So the answer is as simple as setting the initial weight vector: w = np.zeros(X.shape[1]) $\endgroup$ – adriankroeger May 31 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.