# Gradient descent different implementation cause error

We know that we can get closer to the local minimum of the function by descending our argument according to that rule $$w1 = w0 − γ∇f$$

For example I have a linear regression model that depends on $$b, w1$$ $$f(b + w_1) = b + w_1 x_1$$

My error functional is MSE $$MSE = \frac{1}{l} \sum_{i=1}^{l}{(b + w_i x_i - y_i)^2}$$

So basically I can interpret it as following $$MSE = Q(b, w_1) = \frac{1}{1} (b + w_1 x_1 - y_1)^2$$

So now we need to minimize this $$(b + w_1 x_1 - y_1)^2$$

To do that I take partial derivatives from this function

$$\frac{dQ}{db} = 2(b + w_1 x_1 - y_1)$$ $$\frac{dQ}{dw_1} = 2x_1(b + w_1 x_1 - y_1)$$

And now I can have my gradient $$\nabla Q = \begin{bmatrix}2(b + w_1 x_1 - y_1) & 2x_1(b + w_1 x_1 - y_1) \end{bmatrix}$$

# Here is 2 ways of how I see the solution

## First way (does small MSE, good prediction and I have no problems with it):

b = 0
w1 = 0.1

learning_rate = 0.001
iterations = 5000

rows, columns = X_train.shape
for iteration in range(iterations):
for i in range(rows):
dwb = 2 * (b + w1 * X_train[i][0] - y_train[i])
dw1 = 2 * X_train[i][0] * (b + w1 * X_train[i][0] - y_train[i])

b -= learning_rate * dwb
w1 -= learning_rate * dw1


Basically what I do here is I change $$b$$ or $$w1$$ depending on the its direction for every row in dataset

And then I came to the idea, what if it would be more accurately to take the mean gradient and then subtract it and here comes second solution

b = 0
w1 = 0.1

learning_rate = 0.001
iterations = 5000

rows, columns = X_train.shape
for iteration in range(iterations):
for i in range(rows):
dwb = 2 * (b + w1 * X_train[i][0] - y_train[i])
dw1 = 2 * X_train[i][0] * (b + w1 * X_train[i][0] - y_train[i])


• The second approach is somehow like Momentum concept, which takes past gradients into account to smooth out the steps of gradient descent. For example RMSProp (Root Mean Square) optimizer uses momentum in order to normalize gradients using a moving average. But it considers a weight to be able to tuning the effect of past gradients. Jul 4 at 10:40