Why my cost function is so high?

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

X = boston['LSTAT']
y = boston['MEDV']

def linearRegression2(b0 = 0, b1 = 1, lr = 0.00001, n_times = 1000):
partial_B0 = partial_B1 = 0
errors = []
m = len(y)
i = 0
for i in range(n_times):
error = y - (b0 + X * b1)
#Updating b0 and b1
partial_B0 += -lr * (error.sum() / m)
partial_B1 += -lr * (X.T.dot(error) / m)
b0 = b0 - (lr * partial_B0)
b1 = b1 - (lr * partial_B1)
errors.append(mean_squared_error(y, (b0 + X * b1)))
i += 1

print(b0, b1)
print('Iteration i', i)
return errors

error = linearRegression2()
plt.plot(np.arange(1, 1000+1), error)
plt.ylabel('Error')


However, when I plot my cost function, it is decreasing but still too high. I tried to decrease the learning rate, increase the iteration but not changing much.

• If you want to decrease the loss faster you should increase the learning rate instead of decreasing it. Feb 9, 2020 at 15:32
• I tried that one too. Still not decreasing much. Feb 9, 2020 at 15:33
• One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1. Feb 9, 2020 at 16:28
• That was the problem! Thank you so much! Feb 9, 2020 at 16:39

One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1.