In this paper, on page 5, we find the formula


I am really struggling to understand what is meant by this formula. I think at least some of the following are true:

  1. We're dealing with a linear neural network, i.e. no activation functions.
  2. $z^i$ is the output of one layer of the network, so for the very first hidden layer (closest to the input) of this network we would have: $$Var(y)=nVar(x)Var(W)$$ where $y$ is the output vector of the first hidden layer, $x$ is the input vector and $W$ is the matrix of weights connecting that first layer to the input.

However, it's entirely unclear to me what these variances are. I thought maybe $Var(y)$ just meant the empirical variance of the vector $y$, i.e. the sum of the squares of the differences of the elements of $y$ from the mean of $y$, and likewise for $Var(x)$ and $Var(W)$, where the latter is just the variance of all of the entries of $W$. But under this interpretation the formula turns out to be false numerically, so I'm at a bit of a loss to understand what this equation is supposed to mean.


1. Meaning of the formula


This formula expresses the variance of the values of the activation for each neuron $k$ of the layer $i \rightarrow z^i_k$. This value, under the assumptions that are going to be mentioned along this post, is the same for all the neurons in the layer $i$. This is why the authors express it as $Var(z^i)$ instead of a particular $Var(z^i_k)$. Just to simplify the notation

Note that this value is scalar, as well as the other variances involved in the equation: $Var(x)$ and $Var(W^i)$. So, just as a summary, these variances have the next meanings:

  • $Var(W^i)\rightarrow$ The variance of the random initialization of the weights.
  • $Var(x)\rightarrow$ The variance of each feature (which is asummed to be the same for every feature as we are going to see in the next lines).
  • $Var(z^i)\rightarrow$ As said, this is the variance of the activation of each neuron of the layer $i$.

So now let's see how to reach this formula.

2. Reaching the formula

2.1 Analysing some parts of the paper

Let's analyse some parts of that page of the article that answer the other questions and that are going to be also useful to understand the assumptions that are being made.

For a dense artificial neural network using symmetric activation function $f$ with unit derivative at $0$ (i.e. $f'(0) = 1$), if we write $z^i$ for the activation vector of layer $i$, and $s^i$ the argument vector of the activation function at layer $i$, we have: $$ s^i = z^iW^i + b^i\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,z^{i+1}=f(s^i)$$

We can draw some conclusions given that paragraph:

  • The authors are considering an activation function subject to $f'(0) = 1$ in that part of the article. The fact that $f'(0) = 1$ means that locally the function $f(s^i)$ will behave linearly at the neighborhood of $s^i_k=0\rightarrow f'(s^i_k) \approx 1$, where $k$ is the index of the neuron.
  • As you said $\rightarrow z^i$ represents the activation output of one layer

The variances will be expressed with respect to the input, outpout and weight initialization randomness

This means that the variances that they are going to use are going to be $Var(x)$, $Var(z^i)$ and $Var(W^i)$ respectively, and during the initialization (the weights are only spread randomly at the initialization).

The variance of the biases is not being considered as they are assumed to be initialized to the same value at the initialization $ \Rightarrow Var(b^i)=0$.

Consider the hypothesis that we are in a linear regime at the initialization, that the weights are initialized independently and that the inputs features variances are the same ($=Var(x))$.

Given this, we know that:

  • Linear regime $\Rightarrow f'(s^i_k) \approx 1$ as we saw above.
  • Weights are initialized independently $\Rightarrow Var(W^iW^{i+1})=Var(W^i)Var(W^{i+1})$
  • Inputs features variances are the same $\Rightarrow Var(x_k) = Var(x_{k+1}) \,\,\forall\,\, k$ where $k$ is the index of the input

2.2 Example

So now, let's compute for example the Variance of the activations of the first hidden layer ($i=1$) i.e. $Var(z^1_k)$:

$$ \begin{align} Var(z^1_k) &= Var(f(s^0_k)) && (1)\\\\ &\approx (f'(\mathbb{E}(s^0_k))^2\,\,Var(s^0_k) && (2)\\\\ &= (f'(0))^2\,\,Var(s^0_k) && (3)\\\\ &= Var(s^0_k) && (4) \\\\ &= Var(W^0_k x + b_k^0) && (5)\\\\ &= Var(W^0_k x) = Var(w^0_{k1} x_1 + w^0_{k2} x_2 +... ) && (6)\\\\ &= Var(w^0_{k1} x_1) + Var( w^0_{k2} x_2) +... && (7)\\\\ &= n_0Var(w^0_{kj})Var(x_j) && (8)\\\\ &= n_0Var(W^0)Var(x) && (9) \end{align}$$

Note that we would end up with the same expression of $Var(z^i_k)$ for every neuron $k$ of the layer $i \rightarrow $ Now we understand why $Var(z^i)$ represents the variance of the activation for each neuron in the layer $i$ i.e. $Var(z^i) = Var(z^i_k)$.

2.3 Justifications

The justifications for each step are:

  1. $z^{i+1}=f(s^i) \Rightarrow z^{1}=f(s^0)$
  2. The justification to this can be found in the Wikipedia post "Approximating the variance of a function"
  3. $\mathbb{E}(s^0)= \mathbb{E}(W^0x) = \mathbb{E}(W^0)\mathbb{E}(x)=0$. This is because $x$ and $W^i$ are independent and $W^i$ is assumed to be randomly initialized with mean $0$.
  4. Linear regime $\Rightarrow f'(s^i_k) \approx 1$.
  5. $s^i = W^ix + b^i$
  6. Bias values are initialized independently and $Var(b^i)=0$ as we mentioned above.
  7. This is true because $Cov(w^0_{k1} x_1, w^0_{k2} x_2)=0$. Something that is proved at the end of the post.
  8. (and 9) This is because the weights and inputs are independent, and because of the assumptions made by the authors, which are: $$Var(x_j)=Var(x)\,\,\forall\,k \,\,\,\,\,\,\,\text{ and }\,\,\,\,\,\,\, Var(w_{jk})=Var(W)\,\,\forall\,\,k,j$$

3. Conclusion

Extending this reasoning to all the layers we end up with the equation given by the authors: $$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$

Edit regarding the comments The reasoning holds even if the features are not independent. This is because, we can proof that the covariance of the $z^{i+1}_k$ terms in the same layer $i$ is $0$. This also serves as an explantion to the step $(7)$ of the example above.

To see this let's compute $Cov(w_k^iz^i,w_j^iz^i)$ where $w_k^i$ and $w_j^i$ represent the vector of weights related to the neurons $k$ and $j$ of the layer $i$, with $k\neq j$: $$\begin{align} Cov(w_k^iz^i,\,\,w_j^iz^i) &= \mathbb{E}\left[(w_k^iz^i-\mathbb{E}(w_k^iz^i))(w_j^iz^i-\mathbb{E}(w_j^iz^i))\right]\\ &=\mathbb{E}\left[(w_k^iz^i)(w_j^iz^i)\right]\\ &=\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]\\ &=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]\\ &=0^T\,\mathbb{E}\left[z^i\,w_j^iz^i\right]=0 \end{align}$$

Note that we can do $\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]$ because $w_k^i$ is independent of $z^i\,w_j^iz^i$

Then by extending this to the other neurons of the layer, we can confirm that: $Var(W^iz^i) = n_iVar(W^i)Var(z^i)\rightarrow$ We can reach the equation given by the authors.

  • $\begingroup$ Thank you for the detailed breakdown. For this reasoning to work, we have to assume that the inputs (the $x_i$) are independent, right? So in order to iterate through the layers and get the final product-formula, don't we need the outputs of each layer to be independent? But surely they're not, since all the neurons in a given layer are all computed using the same inputs. $\endgroup$
    – Jack M
    Oct 25 '20 at 12:14
  • $\begingroup$ I think maybe we can show that the outputs from a given layer have zero covariance, which would be enough, provided we assume that either the inputs or the randomly initialized weights are zero-mean, but I'm not sure this is the intended reasoning. $\endgroup$
    – Jack M
    Oct 25 '20 at 12:59
  • $\begingroup$ Yes, the reasoning holds because the covariance of the activation outputs terms is $0$. I added this at the end of the post. Thanks for noticing. $\endgroup$
    – Javier TG
    Oct 25 '20 at 13:47
  • $\begingroup$ So if I understand correctly, the whole reasoning goes through even if the inputs features aren't assumed to be uncorrelated? Because even if they're correlated, once you multiply them through the first layer of weights, the resulting weighted sums are uncorrelated due to the weights being independent and zero-mean. $\endgroup$
    – Jack M
    Oct 25 '20 at 15:57
  • 1
    $\begingroup$ The step which I think is more crucial to reach the result is $(7)$ which is true only if $Cov(w_kz_k,\,\,w_jz_j)=0$ (something that we have proved given the assumptions) which implies $Var(w_kz_k + w_jz_j)=Var(w_kz_k)+Var(w_jz_j)$ $\endgroup$
    – Javier TG
    Oct 25 '20 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.