I'm reading the Wikipedia page on backpropagation and have some questions about the following equations:

$$ \frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-1}}{d z^{L-1}}\cdot \frac{d z^{L-1}}{d a^{L-2}} \cdot \ldots \cdot \frac{d a^1}{d z^1} \cdot \frac{\partial z^1}{\partial x} $$

$$ \frac{d C}{d a^L}\circ (f^L)' \cdot W^L \circ (f^{L-1})' \cdot W^{L-1} \circ \cdots \circ (f^1)' \cdot W^1 $$

where $(x,y)$ is the input, output pair. $a^l$ is the activation at layer $l$. $f^l$ is the activation function at layer $l$. $z^l = (f^l)'$

It's my understanding from this post that the activation of a neuron is the value the neuron outputs in response to its input. The activation function is used to compute the activation.

What is the difference between $a^l$ and $f^l$?

For the first equation I think the following are assignments to the variables:

$a^l(x) = \sigma(f(X,W,b))$ where $\sigma : \mathbb R \rightarrow \mathbb R$ potentially the sigmoid function.

$f^l(X,W,b) = W \cdot X + b$ where $X,W \in \mathbb R^d$ and $b \in \mathbb R$.

Is this correct?

If so it seems backpropogation math considers the change in sum of weighted inputs and bias proportional to the neuron's activation?

I don't understand the second equation. I see the weight matrix $W^l$ is multiplied against the activation function $f^l$, which I think is really $f^l(X, W, b) = \sigma(f_b(X,W,b))$ but am confused because the input vector $X$ is not given. Can anyone clear this equation up?



1 Answer 1


There's quite a few places you've misunderstood/mixued up the notation from the wikipedia article. Please go through the article once more carefully.

  1. $(x,y)$ is the input, output pair

    $(x,y)$ is actually the input, target output pair. note the distinction.


  1. $z^l=(f^l)^\prime$

    $z^l$ is not a derivative, its actually the weighted input at layer $l$ , that is, $Wa^{l-1}$


  1. There also seems to be an additional $\sigma$ in your equation. $f$ is already the activation function, which based on your choice of activation functions could be $\sigma$, the sigmoid function. So there is no need to put it twice:


    Should be $a^l(x)=f(Wa^{l-1})$


  1. Another mistake is

    I see the weight matrix $W^l$ is multiplied against the activation function $f^l$

    Its actually multiplied against the derivative of the activation function, denoted by the tick/prime mark $(f^L)^\prime$, evaluated at $a^{l-1}$, the input from the previous layer. The $X$ notation seems to be missing in the article; it would be helpful to yourself to stick to a consistent notation.


Coming to the question of the difference between the activation and the activation function, it is correctly explained in the linked question.

  • $\begingroup$ Thanks. In the article they say $z^l = (f^l)'$ so it's confusing you say $z^l = (W a^{l-1})$, however your comment makes sense. For the second equation should $(f^l) \cdot W^l X$ be more appropriate on the RHS to describe the change in cost? $\endgroup$
    – Nick
    Commented Jun 24, 2023 at 18:41
  • $\begingroup$ Also, in the first equation do you know why $\frac{\partial z^1}{\partial x}$ is the partial on the RHS vs total everywhere else? $\endgroup$
    – Nick
    Commented Jun 24, 2023 at 18:48
  • $\begingroup$ Also a little confused about the first equation, using your substitution, $\frac{\partial W a^{a-1}}{\partial x}$. Specifically why the partial derivative is used vs total $\endgroup$
    – Nick
    Commented Jun 24, 2023 at 19:00
  • $\begingroup$ I think you might have misinterpreted "...as the derivatives (f^l)'(evaluated at z^l)..." to mean that z^l is the derivative. There is no $X$ in the article, so please refrain from using it. a^(l-1) is probably what you want. the total derivative in this case is a matrix, maybe this lecture will help youtube.com/watch?v=i94OvYb6noo $\endgroup$ Commented Jun 29, 2023 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.