I've a very basic question about cost functions. I'm studying gradient descent and there we're using partial differentiation of features "Theta". But isn't the cost function an absolute value for each data point. i.e., $\big(h(x) -y\big)^2$. I'm getting a little confused how we're differentiating this function? I'm trying to build intuition about this. Thanks!
$\begingroup$
$\endgroup$
2
-
$\begingroup$ h is thought of as a function of theta $\endgroup$– Ben Reiniger ♦Commented Jul 16, 2023 at 1:26
-
$\begingroup$ You need a differentiable cost function in order to compute the gradient that tells you the direction of improvement, i.e. how to adjust the weights, $\theta$, such that to improve i.e. reduce the cost. See this answer: datascience.stackexchange.com/a/28699/150390 $\endgroup$– Luca AnzaloneCommented Jul 16, 2023 at 15:04
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
When computing gradients for gradient decent, the chain rule is the key. Back-propagation (which is used in neural networks to compute the gradient) means basically applying the chain rule again and again and again until all elements of the gradient are computed.
In your case, you have a prediction $h(x;\theta)$ and a cost-function $C(\hat{y},y)$.
Note: I added $\theta$ as parameter for $h$, since the values of $\theta$ will influence the output of $h$ and we explicitly want to compute the derivation with respect to $\theta$. I hope this makey things clearer. In short, we can write:
$$\begin{align}
\hat{y}&=h(x;\theta)\\
C(\hat{y},y)&=(\hat{y}-y)^2
\end{align}$$
If you want to compute the derivation with respect to a single parameter $\theta_k$, you get with the chain rule $$\frac{\partial}{\partial\theta_k}C(\hat{y},y)=\frac{\partial}{\partial\hat{y}}C(\hat{y},y)\cdot \frac{\partial\hat{y}}{\partial\theta_k}$$
With your cost function you then get: $$\begin{align} \frac{\partial}{\partial\hat{y}}C(\hat{y},y) &= 2\cdot(\hat{y}-y) = 2\cdot(h(x;\theta)-y)\\ \frac{\partial\hat{y}}{\partial\theta_k} &= \frac{\partial}{\partial\theta_k}h(x;\theta)=\ldots \end{align}$$
Depending on your concrete $h$, you can go on to compute the gradient.
