2
$\begingroup$

In http://neuralnetworksanddeeplearning.com/chap3.html

The author explains that for for a single neuron in neural net, by choosing cost function as cross entropy we can eliminate the derivative of activation function in the gradient term, if the activation function is choose to be sigmoid function.

In the problem, he asks why we cannot eliminate the input to neuron term x in gradient of cost function with respect to weights.

I had the following reasoning, in order to compute gradient of Cost, we use chain rule, and derive cost with respect to activation, and activation with respect to (w * x + b), and the sum with weight.

For a weight Wi, the derivative of (Summation w * x + b) w.r.to Wi is always xi, and the derivative of activation function cannot know that, so it can never eliminate Xi unless its zero.

Or is there any other subtle reasoning?

$\endgroup$
2
$\begingroup$

The main reason is that the chain rule is hidden in your justification. The fact that the inner function is linear w.r.t weights then derivative w.r.t weight will always result into input $x$ will be somewhere in $\dfrac{\partial C}{\partial w}.$

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.