When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?
I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.
Assume the original state is $(0.1, 0.1, 0.2, 5.0)$ and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).
So, the original length is $$ \begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1 + 0.2\times 0.2 + 5\times 5)} &= \sqrt{25.06} \approx 5.006.\end{align*}$$
As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$
Applying the compensation gives $5.004 / 0.5 = 10.008.$
This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like DropConnect does).
In my actual test, the state-vector of $192$ elements has length of $0.885$ and the masked vector, with compensation has length of $1.305$