I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per Iteration:

y_hat = X.dot(W) + b

dW = - (2 * (X^T ).dot(Y - Y_hat)) / m # how does the minus and matrix multiplications are used instead of Summation?
db = - (2 * np.sum(Y - Y_hat)) / m  # np is numpy

W = W - lr * dW  # update weights
b = b - lr * db

I know from the code is that dW is derivative of the weight matrix per iteration, X^T is the Transpose of the X features, Y is original values of Y , Y_hat are predicted values using the formula X.dot(W)+b.

What I want to know is that dW = - (2 * (X^T ).dot(Y - Y_hat)) / m. Even with the MSE loss, as given in this link in the equation 1.4, it should be something else.

Can someone please elaborate how are the values of dW,db are calculated here?

Whole Python Code is giveb here for Linear Regression


1 Answer 1


dW and db are simply the derivative of the loss function with regards to the weights and biases. Given the loss function

$J = \frac{1}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))^2$

the derivatives of the loss to the weights (dW) and bias are equal to

$\frac{\partial}{\partial W} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i)) * x_i$

$\frac{\partial}{\partial b} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))$

As you can see, these equations align with the code you provided. Equation 1.4 from the link you provided is for $\theta_0$, i.e. the bias, which is the same as the code for db (with the only difference being the minus sign, causing by the fact that $y_i$ and $h(x_i)$ are swapped in the loss function between the first and second article).

  • $\begingroup$ Oh Yeah1 Got it. Thanks. $\endgroup$
    – Deshwal
    Commented Feb 27, 2021 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.