Derivation of the cross-entropy equation in Michael Nielsen's book

Question

I am reading the book http://neuralnetworksanddeeplearning.com/chap3.html by Michael Nielsen. So this is a question mostly for the people familiar with the book and understanding the material.

In the equations (71-75) we are trying to find a cost-function $C$ satisfying:

$$\frac{\partial{C}}{\partial{\omega_j}}=x_j (a-y),$$ $$\frac{\partial{C}}{\partial{b}}=(a-y),$$

where $w_j$ and $b$ are weights and bias of a neuron, $a$ is the output of the sigmoid function

$$a = \sigma(\sum{\omega_{j} x_j + b)}=\sigma(z).$$

We apply the chain rule (equation 73):

$$\frac{\partial{C}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\frac{\partial{a}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\sigma\prime(z).$$

And in the next line the author writes

Using $\sigma\prime(z)=\sigma(z)(1-\sigma(z))=a(1-a)$ the last equation becomes...

Where does this expression come from? $\sigma(z)(1-\sigma(z))$

Answer 1

The author has taken activation function as sigmoid in this case. The derivative of this function can be re-written like

$$\sigma(z) = \frac{1}{1+e^{-x}}$$

whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of chap1.