I trying to implement gradient descent in Python and I am following andrew ng course in order to follow the math. However, my implementation isnt working as I expect it to. It would be great you the community can help me identify my mistake.
as I increase the range from 3 to higher number, I dont converge rather thetas move from very positive to very negative and finally nan because they get extremely small.
following is the code.
import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_boston from sklearn.linear_model import LinearRegression X = pd.DataFrame(load_boston().data, columns = load_boston().feature_names) X['theta0'] = 1 y = load_boston().target y = pd.DataFrame(y, columns = ['target']) theta = pd.DataFrame(np.random.randn(X.shape),columns = ['target'], index = X.columns.values) print('theta shape',theta.shape) print('X shape',X.shape) print('y shape',y.shape) print(theta) def predict(X,theta, ycol = 'target'): return X.dot(theta) mse_values = alpha = 0.01 for i in range(10000): error = predict(X,theta) - y theta = theta - ((alpha)* (1/len(X)) * X.T.dot(error)) mse= np.sum(error**2)/len(X) print('mse: ', mse.values) mse_values.append(mse) print('+'*5) plt.plot(mse_values) plt.show()