# Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example

>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)


it makes sense that the variance is close to 512 because each one of 500000, is a dot product of a 512 vector and a 512 vector, that is sampled from a distribution with a standard deviation of 1 and mean of 0

However, I wanted the variance to go down to 1, and consequently the std to be 1 since standard deviation is square root of variance, where 1 is the variance.

To do this I tried the below

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1/512)
>>> (x @ w).var()
tensor(0.0021)


However the variance is actually now 512 / 512 / 512 instead of 512/ 512

In order to do this correctly, I needed to try

>>> x = t.randn(512)
>>> w = t.randn(512, 500000) * (1 / (512 ** .5))
>>> (x @ w).var()
tensor(1.0216)


Why is that the case?

Basically, because the variance is quadratic in w.

You can consider your problem as computing the variance over scalar products $$z = \vec{x}\vec{w}$$ by drawing 500000 samples of $$\vec{w}$$ from $$p(\vec{w})$$. Let's compute the mean first

$$E_{p(\vec{w})} [ \vec{w}^T\vec{x} ] = \vec{\mu}_w^T \vec{x} = 0$$

because you draw the $$\vec{w}$$ from a zero-centered Gaussian. With the expectation value as zero, the variance is given by

$$E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\ = E_{p(\vec{w})} [ \vec{x}^T\vec{w} \vec{w}^T \vec{x} ] \\ = \vec{x}^T E_{p(\vec{w})} [ \vec{w} \vec{w}^T ] \vec{x}$$

$$E_{p(\vec{w})} [ \vec{w} \vec{w}^T ]$$ is the covariance matrix of the $$\vec{w}$$ and since they stem from independent standard normal distributions is diagonal with $$\sigma_w^2 = 1$$ on the diagonal. So that gives

$$E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] = \vec{x}^T \vec{x} \approx 512$$

as you are approximating the sum of variances of 512 independent samples of standard Gaussian. If you now scale the $$\vec{w}$$ with a constant $$k$$, you can do this calculation again and it will give you

$$E_{p(\vec{w})} [ (\frac{1}{k}\vec{w}^T\vec{x})^2 ] \\ = \frac{1}{k^2} E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\ \approx \frac{1}{k^2} 512$$

So in order to end up with 1 in the very end, you need to scale with $$k=\sqrt{512}$$.

Please see Variance of product of multiple independent random variables to understand the product (matmul) of D dimensional normal distributions becomes a normal distribution of variance D, hence we need to divide $$\frac{1}{\sqrt(D)}$$ to keep the variance as 1.0.