# Dropout dividing by compensation term = overshoots the result?

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?

I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.

Assume the original state is $(0.1, 0.1, 0.2, 5.0)$ and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).

So, the original length is \begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1 + 0.2\times 0.2 + 5\times 5)} &= \sqrt{25.06} \approx 5.006.\end{align*}

As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$

Applying the compensation gives $5.004 / 0.5 = 10.008.$

This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like DropConnect does).

In my actual test, the state-vector of $192$ elements has length of $0.885$ and the masked vector, with compensation has length of $1.305$

Your example is cherry-picked: You mask out small numbers and keep a large one. But dropout is applied randomly. Each of the following six masks, and of the corresponding values for the vector length, is equally likely to appear: \begin{align*} &(1, 1, 0, 0): &\sqrt{0.1^2 + 0.1^2} &\approx 0.1414,\\ &(1, 0, 1, 0): &\sqrt{0.1^2+ 0.2^2} &\approx 0.2236,\\ &(1, 0, 0, 1): &\sqrt{0.1^2+ 5^2} &\approx 5.0010,\\ &(0, 1, 1, 0): &\sqrt{0.1^2 + 0.2^2} &\approx 0.2236,\\ &(0, 1, 0, 1): &\sqrt{0.1^2 + 5^2} &\approx 5.0010,\\ &(0, 0, 1, 1): &\sqrt{0.2^2 + 5^2} &\approx 5.0040.\\ \end{align*} The average vector length is $$\frac16 (0.1414+0.2236+5.0010+0.2236+5.0010+5.0040) = 2.5991,$$ which is roughly half of the original vector length $5.006$. So it makes sense to divide it by the dropout rate of $50\%$.
• Ok, but then we will have $\frac{2.5991}{0.5} = 5.1982$ I would still see it as unreliable - in my question above, I have 192 elements and easility overshoot by 1/3rd. Would it be better to scale-up the masked vector to match the original vector's magnitude (instead of dividing by a fixed compensation)? That way we would always have same magnitude as before, not overshot / undershot – Kari Mar 1 '18 at 19:19