0
$\begingroup$

I am reading the book http://neuralnetworksanddeeplearning.com/chap3.html by Michael Nielsen. So this is a question mostly for the people familiar with the book and understanding the material.

In the equations (71-75) we are trying to find a cost-function $C$ satisfying:

$$\frac{\partial{C}}{\partial{\omega_j}}=x_j (a-y),$$ $$\frac{\partial{C}}{\partial{b}}=(a-y),$$

where $w_j$ and $b$ are weights and bias of a neuron, $a$ is the output of the sigmoid function

$$a = \sigma(\sum{\omega_{j} x_j + b)}=\sigma(z).$$

We apply the chain rule (equation 73):

$$\frac{\partial{C}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\frac{\partial{a}}{\partial{b}}=\frac{\partial{C}}{\partial{a}}\sigma\prime(z).$$

And in the next line the author writes

Using $\sigma\prime(z)=\sigma(z)(1-\sigma(z))=a(1-a)$ the last equation becomes...

Where does this expression come from? $\sigma(z)(1-\sigma(z))$

$\endgroup$
5
$\begingroup$

The author has taken activation function as sigmoid in this case. The derivative of this function can be re-written like

$$\sigma(z) = \frac{1}{1+e^{-x}}$$

whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of chap1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.