I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the community can help me to identify my mistake.

When I increase the range from 3 to higher number, it does not converge, rather thetas move from very positive to very negative and finally get nan because they get extremely small.

Code is given below:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
X = pd.DataFrame(load_boston().data, columns = load_boston().feature_names)
X['theta0'] = 1
y = load_boston().target
y = pd.DataFrame(y, columns = ['target'])
theta = pd.DataFrame(np.random.randn(X.shape[1]),columns = ['target'], index = X.columns.values)
print('theta shape',theta.shape)
print('X shape',X.shape)
print('y shape',y.shape)
def predict(X,theta, ycol = 'target'):
    return X.dot(theta)
mse_values =[]
alpha = 0.01
for i in range(10000):
   error = predict(X,theta) - y
   theta = theta - ((alpha)* (1/len(X)) * X.T.dot(error))
   mse= np.sum(error**2)/len(X)
   print('mse: ', mse.values)
  • $\begingroup$ Kindly verify with many available implementations! $\endgroup$
    – Aditya
    Commented Nov 3, 2018 at 15:34
  • $\begingroup$ @Aditya thats exactly the problem, I did but couldnt find anything wrong $\endgroup$ Commented Nov 3, 2018 at 15:39
  • $\begingroup$ take a look at this implementation: bit.ly/2QhuRXN $\endgroup$ Commented Nov 3, 2018 at 15:40
  • 1
    $\begingroup$ Play with your alpha and iter more maybe $\endgroup$
    – Aditya
    Commented Nov 3, 2018 at 15:41

2 Answers 2


If you use the backtracking method (details in my answer in this link:

Does gradient descent always converge to an optimum?)

then you can avoid spending time to manually find the "right learning rate" as in your case here.


I was doubting my implementation all the way but it was the learning rate. After a lot of experimentation, I found the right one, but I'm very much surprised to see how small the learning rate had to be in order for it to work, i.e alpha = 0.000001

  • $\begingroup$ It can be really easy to just assume something else is wrong. Sometimes, using batch and normalization helps as well. $\endgroup$ Commented Nov 4, 2018 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.