Im trying to implement following gradient descent function in Python for logistic regression:

$∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$

This is my python implementation:

def gradient(X, y, theta):
    dtheta = -(np.dot(X.T,y - np.exp(X * theta)))
    return dtheta

X is a dataframe of size: (2458, 31), y is a dataframe of size: (2458, 1) theta is dataframe of size: (2458,1)

when i pass values to my gradient descent function, it returns a dtheta parameter with size (31,31) due to which i cannot update my theta to pass it to cost function, i cannot fig out where im going wrong. any help will be appreciated.

Error i keep getting is: ValueError: operands could not be broadcast together with shapes (2458,1) (31,31)

and this is how im implementing the algorithm:

theta = np.random.uniform(low=-0.1,high=0.1, size=(2458,1))
# Iterate and update theta by using the gradient of the negative log-likelihood
max_iter = 100
learning_rate = 1e-3
for i in range(max_iter):
    # Calculate the gradient
    dtheta = gradient(X,y,theta)

    # Update theta

    theta = (theta - learning_rate) * dtheta

    # Calculate the value of the log-likelihood
    cost = negative_loglikelihood(X,y,theta)

    # Print iteration
    print("Iteration %d, cost function %.3f" % (i+1,cost))
  • $\begingroup$ Can you share the source of the above gradient descent formula? $\endgroup$ Nov 4 '19 at 5:22
  • $\begingroup$ This is part of an assignment that i received, in which i have a negative log likelihood function whose gradient function is as above. $\endgroup$ Nov 4 '19 at 5:26

Check your theta dimensions.

Most likely, your X dimensions indicate you have 2458 training samples per iteration with each having 31 features. Hence, your theta should be a matrix of shape (31, 1).

With X having shape (2458, 31) and if theta has shape (31, 1), X*theta will have dimensions (2458, 1), same as y and as expected. Now, y-theta has the same dimensions as that of y or theta. And so does exp(y-theta)

X_T has shape (31, 2458) and hence, d_theta = - X_T*exp(y-theta) will have shape (31, 1), same as our initial assumed theta shape and now, you can subtract d_theta from theta.

  • $\begingroup$ Thank you very much, that seems to have solved the issue. Though now im facing the similar issue in implementing my neg. log likelihood function,I may have implemented it wrong, $\endgroup$ Nov 4 '19 at 6:13
  • $\begingroup$ Consider upvoting and accepting the answer :) $\endgroup$ Nov 4 '19 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.