Eliminate input in gradient by clever choosing of cost function in neural networks

Question

In http://neuralnetworksanddeeplearning.com/chap3.html

The author explains that for for a single neuron in neural net, by choosing cost function as cross entropy we can eliminate the derivative of activation function in the gradient term, if the activation function is choose to be sigmoid function.

In the problem, he asks why we cannot eliminate the input to neuron term x in gradient of cost function with respect to weights.

I had the following reasoning, in order to compute gradient of Cost, we use chain rule, and derive cost with respect to activation, and activation with respect to (w * x + b), and the sum with weight.

For a weight Wi, the derivative of (Summation w * x + b) w.r.to Wi is always xi, and the derivative of activation function cannot know that, so it can never eliminate Xi unless its zero.

Or is there any other subtle reasoning?

Answer 1

The main reason is that the chain rule is hidden in your justification. The fact that the inner function is linear w.r.t weights then derivative w.r.t weight will always result into input $x$ will be somewhere in $\dfrac{\partial C}{\partial w}.$