0
$\begingroup$

I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far:

import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error


dataset = load_boston()
X = boston['LSTAT']
y = boston['MEDV']

def linearRegression2(b0 = 0, b1 = 1, lr = 0.00001, n_times = 1000):
partial_B0 = partial_B1 = 0
errors = []
m = len(y)
i = 0
for i in range(n_times):
    error = y - (b0 + X * b1)
    #Updating b0 and b1
    partial_B0 += -lr * (error.sum() / m) 
    partial_B1 += -lr * (X.T.dot(error) / m)
    b0 = b0 - (lr * partial_B0)
    b1 = b1 - (lr * partial_B1)
    errors.append(mean_squared_error(y, (b0 + X * b1)))
    i += 1

print(b0, b1)
print('Iteration i', i)
return errors


error = linearRegression2()
plt.plot(np.arange(1, 1000+1), error)
plt.ylabel('Error')

However, when I plot my cost function, it is decreasing but still too high. I tried to decrease the learning rate, increase the iteration but not changing much.

enter image description here

$\endgroup$
4
  • $\begingroup$ If you want to decrease the loss faster you should increase the learning rate instead of decreasing it. $\endgroup$
    – Oxbowerce
    Feb 9, 2020 at 15:32
  • $\begingroup$ I tried that one too. Still not decreasing much. $\endgroup$ Feb 9, 2020 at 15:33
  • $\begingroup$ One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1. $\endgroup$
    – Oxbowerce
    Feb 9, 2020 at 16:28
  • $\begingroup$ That was the problem! Thank you so much! $\endgroup$ Feb 9, 2020 at 16:39

1 Answer 1

1
$\begingroup$

One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.