1
$\begingroup$

In the online book on neural networks by Michael Nielsen, in chapter 3, he introduces a new cost function called as log-likelihood function defined as below
$$ C = -ln(a_y^L) $$ Suppose we have 10 output neurons, when back propagating the error, only the gradient w.r.t. $y^{th}$ output neuron is non-zero and all others are zero. Is that right?

If so, how is the below equation (81) true? $$\frac{\partial C}{\partial b_j^L} = a_j^L - y_j $$ I'm getting the expression as $$\frac{\partial C}{\partial b_j^L} = y_j (a_j^L - 1) $$

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

No you actually didn't really understand how softmax functions it outputs a probability distribution hence if there are 10 output neurons you will have 10 probabilities for the 10 respective classes i.e. the neuron with the highest probability will be more activated that is none of the output neurons will give 0 as output that is what softmax is it takes exponential average of every class to produce a probability distribution over k different classes here k=10.Now as you said suppose you have 10 output neurons then while back propagating the error, only the gradient w.r.t. yth output neuron is non-zero and all others are zero,this is wrong if you go and give it a read the error or cost function is calculated as follows when there are multiple neurons: enter image description here

Now as you can see the cost is calculated over all the output neurons for all the training examples in the batch that is of size 'n' here hence when you do backpropagation that is when you calculate"dc/dw" and "dc/db" it includes output from all the output neurons and your statement is wrong because if gradients of other output neurons are 0 then how will your backpropagation update there weight matrices. I know its confusing but if you read his ch-2 you should be able to understand it.i have used cross entropy to explain you this but the same method will work for any cost function that you take .

As far as the derivation is considered it is quite easy just go back and study his chapter two all four equation BP1,BP2,BP3,BP4 and understand there derivation it will take some time but it is easy once you understand the composite function nature of neural networks and how to differentiate composite functions using the chain rule.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I never said that other activations are zero. I said gradients w.r.t. other activations are zero. This is because the cost function depends on only one activation term, not all. That is my confusion. $\endgroup$
    – Nagabhushan S N
    Aug 19, 2019 at 2:39
  • $\begingroup$ the value of cost function doesn't depend only on the activated neuron's activation, what you see as output are probabilities which are rounded off so that you can get your result to as to which class that instance of your dataset belonged to, but during backprop output activations from all the neurons are summed to calculate the actual value of loss function, hence your statement that gradients w.r.t to other activations = 0 is wrong. $\endgroup$
    – khwaja wisal
    Dec 10, 2019 at 19:35
  • $\begingroup$ Okay. So, what is the actual cost function? Can you provide the expression for that? $\endgroup$
    – Nagabhushan S N
    Dec 11, 2019 at 1:51
  • $\begingroup$ @NagabhushanSN, I gave you the expression when i answered the question, the first time, the above equation is the cross-entropy loss function, the first summation represents the number of samples in the batch(x) and the second summation sums the activation across all the neurons of the output layer(j represents neurons in the output layer). $\endgroup$
    – khwaja wisal
    Jan 24, 2020 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.