# Derivative of MSE Cost Function

$$\theta_{t+1}=\theta_t-a\frac{\partial}{\partial \theta_j}J(\theta)$$

But specifically about $$J$$ cost function (Mean Squared Error) partial derivative:

Consider that: $$h_\theta(x)=\theta_0+\theta_1x$$

$$\frac{\partial}{\partial\theta_j}J(\theta) = \frac{\partial}{\partial\theta_j}\frac{1}{2}(h_{\theta}(x)-y)^2$$

$$\ \ \ \ \ \ \ \ \ \ \ \ =2\frac{1}{2}(h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(h_{\theta}(x)-y)$$

$$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)*\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y_i)$$

$$\ \ \ \ \ \ \ \ \ \ \ \ = (h_{\theta}(x)-y)x_j$$

It´s not clear to me how $$x_j$$ is calculated:

$$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j$$

Can anyone help me to understand in detail this part of the partial derivative? Thanks in advance.

• Welcome to Data Science SE. Could you please link the source material and explain context? The answer is relatively simple though . . . Commented May 17, 2022 at 7:13
• Thanks for answering. @NeilSlater, the context is the process of training an artificial neural network, specifically when weights are updated. The best weights minimizes a given cost function (J(θ) in my question). en.wikipedia.org/wiki/Stochastic_gradient_descent Commented May 17, 2022 at 13:35
• Thanks for the update. I think the derivation you have given does not work for all the params of a general neural network, but only for the last layer, or in the case of linear regression. Commented May 17, 2022 at 13:39

Any term $$f$$ that is not a function of $$\theta_j$$ in any equation will have a partial derivative $$\frac{\partial}{\partial\theta_j}(f) = 0$$. Importantly, no $$x_i$$, $$y$$ or $$\theta_{i \ne j}$$ depend in any way upon $$\theta_j$$, so they are effectively constants when figuring out the partial derivative. This is also true for any function of them, provided that also does not depend on $$\theta_j$$.
So for example $$\frac{\partial}{\partial\theta_j}(f(y)) = 0$$, $$\frac{\partial}{\partial\theta_j}(f(y)\theta_j) = f(y)$$ and $$\frac{\partial}{\partial\theta_j}(f(y)\theta_j^2) = 2f(y)\theta_j$$
$$\frac{\partial}{\partial\theta_j}(\sum_{i=0}^{n}\theta_ix_i-y) = x_j$$
When $$i \ne j$$, then $$\frac{\partial}{\partial\theta_j}(\theta_ix_i-y) = 0$$, because no term inside the brackets depends on $$\theta_j$$.
When $$i = j$$, then $$\frac{\partial}{\partial\theta_j}(\theta_jx_j-y) = x_j$$. Only the term $$\theta_jx_j$$ depends on $$\theta_j$$, and it is a linear multiplication.