1
$\begingroup$

The gradient descent:

$\theta_{t+1}=\theta_t-a\frac{\partial}{\partial \theta_j}J(\theta)$

But specifically about $J$ cost function (Mean Squared Error) partial derivative:

Consider that: $h_\theta(x)=\theta_0+\theta_1x$

$\frac{\partial}{\partial\theta_j}J(\theta) = \frac{\partial}{\partial\theta_j}\frac{1}{2}(h_{\theta}(x)-y)^2$

$\ \ \ \ \ \ \ \ \ \ \ \ =2\frac{1}{2}(h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(h_{\theta}(x)-y)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y_i)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)x_j$

It´s not clear to me how $x_j$ is calculated:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

Can anyone help me to understand in detail this part of the partial derivative? Thanks in advance.

$\endgroup$
3
  • $\begingroup$ Welcome to Data Science SE. Could you please link the source material and explain context? The answer is relatively simple though . . . $\endgroup$ May 17, 2022 at 7:13
  • $\begingroup$ Thanks for answering. @NeilSlater, the context is the process of training an artificial neural network, specifically when weights are updated. The best weights minimizes a given cost function (J(θ) in my question). en.wikipedia.org/wiki/Stochastic_gradient_descent $\endgroup$ May 17, 2022 at 13:35
  • $\begingroup$ Thanks for the update. I think the derivation you have given does not work for all the params of a general neural network, but only for the last layer, or in the case of linear regression. $\endgroup$ May 17, 2022 at 13:39

1 Answer 1

3
$\begingroup$

Any term $f$ that is not a function of $\theta_j$ in any equation will have a partial derivative $\frac{\partial}{\partial\theta_j}(f) = 0$. Importantly, no $x_i$, $y$ or $\theta_{i \ne j}$ depend in any way upon $\theta_j$, so they are effectively constants when figuring out the partial derivative. This is also true for any function of them, provided that also does not depend on $\theta_j$.

So for example $\frac{\partial}{\partial\theta_j}(f(y)) = 0$, $\frac{\partial}{\partial\theta_j}(f(y)\theta_j) = f(y)$ and $\frac{\partial}{\partial\theta_j}(f(y)\theta_j^2) = 2f(y)\theta_j$

From this:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

When $i \ne j$, then $\frac{\partial}{\partial\theta_j}(\theta_ix_i-y) = 0$, because no term inside the brackets depends on $\theta_j$.

When $i = j$, then $\frac{\partial}{\partial\theta_j}(\theta_jx_j-y) = x_j$. Only the term $\theta_jx_j$ depends on $\theta_j$, and it is a linear multiplication.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.