Neural Nets: Difference between activation and activation function, error on Wikipedia?

Question

I'm reading the Wikipedia page on backpropagation and have some questions about the following equations:

$$ \frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-1}}{d z^{L-1}}\cdot \frac{d z^{L-1}}{d a^{L-2}} \cdot \ldots \cdot \frac{d a^1}{d z^1} \cdot \frac{\partial z^1}{\partial x} $$

$$ \frac{d C}{d a^L}\circ (f^L)' \cdot W^L \circ (f^{L-1})' \cdot W^{L-1} \circ \cdots \circ (f^1)' \cdot W^1 $$

where $(x,y)$ is the input, output pair. $a^l$ is the activation at layer $l$. $f^l$ is the activation function at layer $l$. $z^l = (f^l)'$

It's my understanding from this post that the activation of a neuron is the value the neuron outputs in response to its input. The activation function is used to compute the activation.

What is the difference between $a^l$ and $f^l$?

For the first equation I think the following are assignments to the variables:

$a^l(x) = \sigma(f(X,W,b))$ where $\sigma : \mathbb R \rightarrow \mathbb R$ potentially the sigmoid function.

$f^l(X,W,b) = W \cdot X + b$ where $X,W \in \mathbb R^d$ and $b \in \mathbb R$.

Is this correct?

If so it seems backpropogation math considers the change in sum of weighted inputs and bias proportional to the neuron's activation?

I don't understand the second equation. I see the weight matrix $W^l$ is multiplied against the activation function $f^l$, which I think is really $f^l(X, W, b) = \sigma(f_b(X,W,b))$ but am confused because the input vector $X$ is not given. Can anyone clear this equation up?

Thanks

Answer 1

There's quite a few places you've misunderstood/mixued up the notation from the wikipedia article. Please go through the article once more carefully.

1. $(x,y)$ is the input, output pair

   $(x,y)$ is actually the input, target output pair. note the distinction.

 

2. $z^l=(f^l)^\prime$

   $z^l$ is not a derivative, its actually the weighted input at layer $l$ , that is, $Wa^{l-1}$

 

3. There also seems to be an additional $\sigma$ in your equation. $f$ is already the activation function, which based on your choice of activation functions could be $\sigma$, the sigmoid function. So there is no need to put it twice:

   $a^l(x)=\sigma(f(X,W,b))$

   Should be $a^l(x)=f(Wa^{l-1})$

 

4. Another mistake is

   I see the weight matrix $W^l$ is multiplied against the activation function $f^l$

   Its actually multiplied against the derivative of the activation function, denoted by the tick/prime mark $(f^L)^\prime$, evaluated at $a^{l-1}$, the input from the previous layer. The $X$ notation seems to be missing in the article; it would be helpful to yourself to stick to a consistent notation.

 

Coming to the question of the difference between the activation and the activation function, it is correctly explained in the linked question.