0
$\begingroup$

first look at this example

>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)

it makes sense that the variance is close to 512 because each one of 500000, is a dot product of a 512 vector and a 512 vector, that is sampled from a distribution with a standard deviation of 1 and mean of 0

However, I wanted the variance to go down to 1, and consequently the std to be 1 since standard deviation is square root of variance, where 1 is the variance.

To do this I tried the below

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1/512)
>>> (x @ w).var()
tensor(0.0021)

However the variance is actually now 512 / 512 / 512 instead of 512/ 512

In order to do this correctly, I needed to try

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1 / (512 ** .5))
>>> (x @ w).var()
tensor(1.0216)

Why is that the case?

$\endgroup$

2 Answers 2

2
$\begingroup$

Basically, because the variance is quadratic in w.

You can consider your problem as computing the variance over scalar products $z = \vec{x}\vec{w}$ by drawing 500000 samples of $\vec{w}$ from $p(\vec{w})$. Let's compute the mean first

$$ E_{p(\vec{w})} [ \vec{w}^T\vec{x} ] = \vec{\mu}_w^T \vec{x} = 0$$

because you draw the $\vec{w}$ from a zero-centered Gaussian. With the expectation value as zero, the variance is given by

$$ E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\ = E_{p(\vec{w})} [ \vec{x}^T\vec{w} \vec{w}^T \vec{x} ] \\ = \vec{x}^T E_{p(\vec{w})} [ \vec{w} \vec{w}^T ] \vec{x} $$

$E_{p(\vec{w})} [ \vec{w} \vec{w}^T ]$ is the covariance matrix of the $\vec{w}$ and since they stem from independent standard normal distributions is diagonal with $\sigma_w^2 = 1$ on the diagonal. So that gives

$$ E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] = \vec{x}^T \vec{x} \approx 512 $$

as you are approximating the sum of variances of 512 independent samples of standard Gaussian. If you now scale the $\vec{w}$ with a constant $k$, you can do this calculation again and it will give you

$$ E_{p(\vec{w})} [ (\frac{1}{k}\vec{w}^T\vec{x})^2 ] \\ = \frac{1}{k^2} E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\ \approx \frac{1}{k^2} 512 $$

So in order to end up with 1 in the very end, you need to scale with $k=\sqrt{512}$.

I hope this answers your question.

$\endgroup$
0
$\begingroup$

Please look at Building makemore Part 3: Activations & Gradients, BatchNorm where Andrej Karpathy specifically explains this problem, and how to keep the layer signals to follow normal distribution of variance 1.0.

Please see Variance of product of multiple independent random variables to understand the product (matmul) of D dimensional normal distributions becomes a normal distribution of variance D, hence we need to divide $\frac{1}{\sqrt(D)}$ to keep the variance as 1.0.

Once these are understood, then Xavier weight initialization and He initialization used in neural network will make sense that they indeed address the problem of keeping variance to 1.0, and you would start using them to keep the variance to 1.0. Please note the initialization depends on the activation functions for you to use. Please see TORCH.NN.INIT on how we need to adjust the weight initialization depending on the functions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.