I was trying to code the Linear Regression in Python using Matrix Multiplication method using Gradient Descent and followed a code where there was no mention what is the loss but just a code as Per Iteration:

y_hat = X.dot(W) + b

dW = - (2 * (X^T ).dot(Y - Y_hat)) / m # how does the minus and matrix multiplications are used instead of Summation?
db = - (2 * np.sum(Y - Y_hat)) / m  # np is numpy

W = W - lr * dW  # update weights
b = b - lr * db

I know from the code is that dW is derivative of the weight matrix per iteration, X^T is the Transpose of the X features, Y is original values of Y , Y_hat are predicted values using the formula X.dot(W)+b.

What I want to know is that dW = - (2 * (X^T ).dot(Y - Y_hat)) / m. Even with the MSE loss, as given in this link in the equation 1.4, it should be something else.

Can someone please elaborate how are the values of dW,db are calculated here?

Whole Python Code is giveb here for Linear Regression


dW and db are simply the derivative of the loss function with regards to the weights and biases. Given the loss function

$J = \frac{1}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))^2$

the derivatives of the loss to the weights (dW) and bias are equal to

$\frac{\partial}{\partial W} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i)) * x_i$

$\frac{\partial}{\partial b} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))$

As you can see, these equations align with the code you provided. Equation 1.4 from the link you provided is for $\theta_0$, i.e. the bias, which is the same as the code for db (with the only difference being the minus sign, causing by the fact that $y_i$ and $h(x_i)$ are swapped in the loss function between the first and second article).

  • $\begingroup$ Oh Yeah1 Got it. Thanks. $\endgroup$ – Deshwal Feb 27 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.