0
$\begingroup$

When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?

I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.

Assume the original state is $(0.1, 0.1, 0.2, 5.0)$ and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).

So, the original length is $$ \begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1 + 0.2\times 0.2 + 5\times 5)} &= \sqrt{25.06} \approx 5.006.\end{align*}$$

As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$

Applying the compensation gives $5.004 / 0.5 = 10.008.$

This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like DropConnect does).

In my actual test, the state-vector of $192$ elements has length of $0.885$ and the masked vector, with compensation has length of $1.305$

|improve this question|||||
$\endgroup$
2
$\begingroup$

Your example is cherry-picked: You mask out small numbers and keep a large one. But dropout is applied randomly. Each of the following six masks, and of the corresponding values for the vector length, is equally likely to appear: $$ \begin{align*} &(1, 1, 0, 0): &\sqrt{0.1^2 + 0.1^2} &\approx 0.1414,\\ &(1, 0, 1, 0): &\sqrt{0.1^2+ 0.2^2} &\approx 0.2236,\\ &(1, 0, 0, 1): &\sqrt{0.1^2+ 5^2} &\approx 5.0010,\\ &(0, 1, 1, 0): &\sqrt{0.1^2 + 0.2^2} &\approx 0.2236,\\ &(0, 1, 0, 1): &\sqrt{0.1^2 + 5^2} &\approx 5.0010,\\ &(0, 0, 1, 1): &\sqrt{0.2^2 + 5^2} &\approx 5.0040.\\ \end{align*} $$ The average vector length is $$ \frac16 (0.1414+0.2236+5.0010+0.2236+5.0010+5.0040) = 2.5991, $$ which is roughly half of the original vector length $5.006$. So it makes sense to divide it by the dropout rate of $50\%$.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Ok, but then we will have $\frac{2.5991}{0.5} = 5.1982$ I would still see it as unreliable - in my question above, I have 192 elements and easility overshoot by 1/3rd. Would it be better to scale-up the masked vector to match the original vector's magnitude (instead of dividing by a fixed compensation)? That way we would always have same magnitude as before, not overshot / undershot $\endgroup$ – Kari Mar 1 '18 at 19:19
  • 3
    $\begingroup$ @Kari: Compensating to match the original magnitude might have unwanted side effects, as you are leaking some information about the masked items (and technically the gradients will not be correct, because masked items still contribute). However, the idea is not completely unreasonable IMO - it might be worth a try, to see if it results in stable training. Dropout already works well in practice, so I don't expect you will find a major difference. But you can compare different versions of dropout for training speed and metrics at the end, see what happens . . . $\endgroup$ – Neil Slater Mar 1 '18 at 20:40
  • 1
    $\begingroup$ I have turned your follow-up question into a mathematical problem. I'll edit this answer if I hear something. $\endgroup$ – Elias Strehle Mar 1 '18 at 20:45
  • 1
    $\begingroup$ An answer at last: There is no simple solution that matches the average vector length after dropout with the original vector length. You would have to look at the full input vector (not just its norm), which would be computationally expensive. Dividing by the dropout rate is fast, simple and usually not too incorrect. $\endgroup$ – Elias Strehle Mar 11 '18 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.