Derivative of MSE Cost Function

Question

The gradient descent:

$\theta_{t+1}=\theta_t-a\frac{\partial}{\partial \theta_j}J(\theta)$

But specifically about $J$ cost function (Mean Squared Error) partial derivative:

Consider that: $h_\theta(x)=\theta_0+\theta_1x$

$\frac{\partial}{\partial\theta_j}J(\theta) = \frac{\partial}{\partial\theta_j}\frac{1}{2}(h_{\theta}(x)-y)^2$

$\ \ \ \ \ \ \ \ \ \ \ \ =2\frac{1}{2}(h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(h_{\theta}(x)-y)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y_i)$

$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)x_j$

It´s not clear to me how $x_j$ is calculated:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

Can anyone help me to understand in detail this part of the partial derivative? Thanks in advance.

Answer 1

Any term $f$ that is not a function of $\theta_j$ in any equation will have a partial derivative $\frac{\partial}{\partial\theta_j}(f) = 0$. Importantly, no $x_i$, $y$ or $\theta_{i \ne j}$ depend in any way upon $\theta_j$, so they are effectively constants when figuring out the partial derivative. This is also true for any function of them, provided that also does not depend on $\theta_j$.

So for example $\frac{\partial}{\partial\theta_j}(f(y)) = 0$, $\frac{\partial}{\partial\theta_j}(f(y)\theta_j) = f(y)$ and $\frac{\partial}{\partial\theta_j}(f(y)\theta_j^2) = 2f(y)\theta_j$

From this:

$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j $

When $i \ne j$, then $\frac{\partial}{\partial\theta_j}(\theta_ix_i-y) = 0$, because no term inside the brackets depends on $\theta_j$.

When $i = j$, then $\frac{\partial}{\partial\theta_j}(\theta_jx_j-y) = x_j$. Only the term $\theta_jx_j$ depends on $\theta_j$, and it is a linear multiplication.