0
$\begingroup$

We know that we can get closer to the local minimum of the function by descending our argument according to that rule $$w1 = w0 − γ∇f$$

For example I have a linear regression model that depends on $b, w1$ $$f(b + w_1) = b + w_1 x_1$$

My error functional is MSE $$MSE = \frac{1}{l} \sum_{i=1}^{l}{(b + w_i x_i - y_i)^2}$$

So basically I can interpret it as following $$MSE = Q(b, w_1) = \frac{1}{1} (b + w_1 x_1 - y_1)^2$$

So now we need to minimize this $$(b + w_1 x_1 - y_1)^2$$

To do that I take partial derivatives from this function

$$\frac{dQ}{db} = 2(b + w_1 x_1 - y_1)$$ $$\frac{dQ}{dw_1} = 2x_1(b + w_1 x_1 - y_1)$$

And now I can have my gradient $$\nabla Q = \begin{bmatrix}2(b + w_1 x_1 - y_1) & 2x_1(b + w_1 x_1 - y_1) \end{bmatrix} $$

Here is 2 ways of how I see the solution

First way (does small MSE, good prediction and I have no problems with it):

b = 0
w1 = 0.1

learning_rate = 0.001
iterations = 5000

rows, columns = X_train.shape
for iteration in range(iterations):
    for i in range(rows):
        dwb = 2 * (b + w1 * X_train[i][0] - y_train[i])
        dw1 = 2 * X_train[i][0] * (b + w1 * X_train[i][0] - y_train[i])
        
        b -= learning_rate * dwb
        w1 -= learning_rate * dw1

enter image description here

Basically what I do here is I change $b$ or $w1$ depending on the its direction for every row in dataset

And then I came to the idea, what if it would be more accurately to take the mean gradient and then subtract it and here comes second solution

Second way (very bad, produces huge MSE, bad prediction):

b = 0
w1 = 0.1

learning_rate = 0.001
iterations = 5000

rows, columns = X_train.shape
for iteration in range(iterations):
    grad = np.array([0, 0])
    for i in range(rows):
        dwb = 2 * (b + w1 * X_train[i][0] - y_train[i])
        dw1 = 2 * X_train[i][0] * (b + w1 * X_train[i][0] - y_train[i])
        
        grad = grad + np.array([dwb, dw1])
        
    grad = grad / rows
    b -= learning_rate * grad[0]
    w1 -= learning_rate * grad[1]

enter image description here

So, my question is why does it happen that mean gradient does not work in that way?

$\endgroup$
2
  • $\begingroup$ The second approach is somehow like Momentum concept, which takes past gradients into account to smooth out the steps of gradient descent. For example RMSProp (Root Mean Square) optimizer uses momentum in order to normalize gradients using a moving average. But it considers a weight to be able to tuning the effect of past gradients. $\endgroup$
    – Kaveh
    Jul 4 at 10:40
  • $\begingroup$ @Kaveh, thank you, and I actually figured this out, the problem was that learning rate was very small and 5000 iterations was not enough $\endgroup$ Jul 4 at 10:51
0
$\begingroup$

The problem with second solution is that learning rate was very small and 5000 iterations was not enough! With learning rate = 0.1 and iterations = 10**4 works perfect

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.