# Gradient Descent Python Implementation isnt converging

I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the community can help me identify my mistake.

as I increase the range from 3 to higher number, I dont converge rather thetas move from very positive to very negative and finally nan because they get extremely small.

following is the code.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

X['theta0'] = 1
y = pd.DataFrame(y, columns = ['target'])
theta = pd.DataFrame(np.random.randn(X.shape),columns = ['target'], index = X.columns.values)

print('theta shape',theta.shape)
print('X shape',X.shape)
print('y shape',y.shape)
print(theta)

def predict(X,theta, ycol = 'target'):
return X.dot(theta)

mse_values =[]
alpha = 0.01
for i in range(10000):
error = predict(X,theta) - y
theta = theta - ((alpha)* (1/len(X)) * X.T.dot(error))
mse= np.sum(error**2)/len(X)
print('mse: ', mse.values)
mse_values.append(mse)
print('+'*5)

plt.plot(mse_values)
plt.show()

• Kindly verify with many available implementations! – Aditya Nov 3 '18 at 15:34
• @Aditya thats exactly the problem, I did but couldnt find anything wrong – Shoaibkhanz Nov 3 '18 at 15:39
• take a look at this implementation: bit.ly/2QhuRXN – Shoaibkhanz Nov 3 '18 at 15:40
• Play with your alpha and iter more maybe – Aditya Nov 3 '18 at 15:41

I was doubting my implementation all the way but it was the learning rate. after a lot of experimentation I found the right one, but I am very much surprised as to how small the learning rate had to be in order for it to work, i.e alpha = 0.000001