# Neural Nets: Difference between activation and activation function, error on Wikipedia?

$$\frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-1}}{d z^{L-1}}\cdot \frac{d z^{L-1}}{d a^{L-2}} \cdot \ldots \cdot \frac{d a^1}{d z^1} \cdot \frac{\partial z^1}{\partial x}$$

$$\frac{d C}{d a^L}\circ (f^L)' \cdot W^L \circ (f^{L-1})' \cdot W^{L-1} \circ \cdots \circ (f^1)' \cdot W^1$$

where $$(x,y)$$ is the input, output pair. $$a^l$$ is the activation at layer $$l$$. $$f^l$$ is the activation function at layer $$l$$. $$z^l = (f^l)'$$

It's my understanding from this post that the activation of a neuron is the value the neuron outputs in response to its input. The activation function is used to compute the activation.

What is the difference between $$a^l$$ and $$f^l$$?

For the first equation I think the following are assignments to the variables:

$$a^l(x) = \sigma(f(X,W,b))$$ where $$\sigma : \mathbb R \rightarrow \mathbb R$$ potentially the sigmoid function.

$$f^l(X,W,b) = W \cdot X + b$$ where $$X,W \in \mathbb R^d$$ and $$b \in \mathbb R$$.

Is this correct?

If so it seems backpropogation math considers the change in sum of weighted inputs and bias proportional to the neuron's activation?

I don't understand the second equation. I see the weight matrix $$W^l$$ is multiplied against the activation function $$f^l$$, which I think is really $$f^l(X, W, b) = \sigma(f_b(X,W,b))$$ but am confused because the input vector $$X$$ is not given. Can anyone clear this equation up?

Thanks

There's quite a few places you've misunderstood/mixued up the notation from the wikipedia article. Please go through the article once more carefully.

1. $$(x,y)$$ is the input, output pair

$$(x,y)$$ is actually the input, target output pair. note the distinction.

1. $$z^l=(f^l)^\prime$$

$$z^l$$ is not a derivative, its actually the weighted input at layer $$l$$ , that is, $$Wa^{l-1}$$

1. There also seems to be an additional $$\sigma$$ in your equation. $$f$$ is already the activation function, which based on your choice of activation functions could be $$\sigma$$, the sigmoid function. So there is no need to put it twice:

$$a^l(x)=\sigma(f(X,W,b))$$

Should be $$a^l(x)=f(Wa^{l-1})$$

1. Another mistake is

I see the weight matrix $$W^l$$ is multiplied against the activation function $$f^l$$

Its actually multiplied against the derivative of the activation function, denoted by the tick/prime mark $$(f^L)^\prime$$, evaluated at $$a^{l-1}$$, the input from the previous layer. The $$X$$ notation seems to be missing in the article; it would be helpful to yourself to stick to a consistent notation.

Coming to the question of the difference between the activation and the activation function, it is correctly explained in the linked question.

• Thanks. In the article they say $z^l = (f^l)'$ so it's confusing you say $z^l = (W a^{l-1})$, however your comment makes sense. For the second equation should $(f^l) \cdot W^l X$ be more appropriate on the RHS to describe the change in cost?
– Nick
Jun 24 at 18:41
• Also, in the first equation do you know why $\frac{\partial z^1}{\partial x}$ is the partial on the RHS vs total everywhere else?
– Nick
Jun 24 at 18:48
• Also a little confused about the first equation, using your substitution, $\frac{\partial W a^{a-1}}{\partial x}$. Specifically why the partial derivative is used vs total
– Nick
Jun 24 at 19:00
• I think you might have misinterpreted "...as the derivatives (f^l)'(evaluated at z^l)..." to mean that z^l is the derivative. There is no $X$ in the article, so please refrain from using it. a^(l-1) is probably what you want. the total derivative in this case is a matrix, maybe this lecture will help youtube.com/watch?v=i94OvYb6noo Jun 29 at 8:01