Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

Question

First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-gradient-descent-in-linear-regression, https://stats.stackexchange.com/questions/212619/why-is-gradient-descent-required but couldn't find my answer.

Gradient descent is: $w_{i}:=w_{i}-\alpha \frac{\delta }{\delta w_{i}}j(w)$ where w is a vector.

At his book "Pattern Recognition and Machine Learning" Bishop says:

"Because the error function is a quadratic function of the coefficients w, its derivatives with respect to the coefficients will be linear in the elements of w, and so the minimization of the error function has a unique solution..."

So if we take derivative of $j(w)$ with respect to $w_{i}$ and equal to zero, in the end it will give us the smallest $w$. Which is actually the first exercise.

In gradient descent we take the derivative too, so the problem can't be because of derivative. Such as an equation that its derivative can't be found. If we can find the answer with one iteration(making equation equal to zero for every feature) why do we iterate over and over again instead, which is the case of gradient descent.

Answer 1

If I understand correctly, your question is Why do we solve gradient descent problems iteratively rather than setting the derivative to 0 and solving for the weights which minimize the cost function?

We can't do this (in all but extremely simple problems such as linear regression) because there is no easy closed-form solution when you set the derivative equal to 0.

Let's take a look at a simple example: let's say you have a 1-layer neural net with no bias and a sigmoid activation function. Your network output is $\hat{y} = \sigma(WX)$. If you use L2 loss, your cost function would be $L = (\sigma(WX) - y)^2$. (For simplicity we'll assume that we're only using a batch size of 1).

Let's take the derivative with respect to our weights $W$ and use the chain rule : $\begin{eqnarray*} \frac{dL}{dW} &=& 2(\sigma(WX) - y)\frac{d}{dW}\sigma(WX)\\ &=& 2(\sigma(WX) - y)\sigma'(WX)\frac{d}{dW}(WX)\\ &=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))\frac{d}{dW}(WX)\\ &=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))X\\ \end{eqnarray*}$

At this point, we get a bit stuck. If we set this derivative equal to 0, it still won't be easy to solve for $W$ since $W$ appears 3 times in this equation and can't easily be isolated by itself on one side of the equation.

Since we cannot solve for the cost-minimizing weights directly, instead we use gradient descent. We take the derivative of the cost function to tell us which direction we should change each weight in order to most decrease the cost. The actual size of the update is determined by the learning rate $\alpha$.

Answer 2

Simply: you may not be able to solve the equation after taking the derivative.