3
$\begingroup$

In my understanding of batch normalization, mean and variance are calculated over the entire batch and then added to the population average. This average is then applied to the test set to estimate the mean and variance of the overall test set. However, DropConnect removes connections during training as a form of network regularization. The model calculates batch statistics through a network that is missing usually half its connections. At test time, all connections are used. Shouldn't this affect the mean and variance of the population and/or the test set, throwing the estimated mean and variance off, or will the network stabilize itself over time?

$\endgroup$
1
$\begingroup$

TL;DR

This is an interesting idea and probably best to be tested with your specific problem; however it is generally understood that you will get better results by not using both Dropout (including DropConnect) and Batch-Norm together, given their overall effects during training. Recent evidence/tests. Having said that, I think it would balance out in the end anyway, as you propose.

More considerations

Looking at the explanation of the base implementation, you could be correct in saying that batch normalisation computations of mean and variance could be influenced - but surely it depends in which order you compute your batch statistics?

There is an ongoing debate (see e.g. this question) as to which order to apply layers such as batch-norm, dropout and activations themselves to the weights. There is the argument (pointed out above) that using both together isn't a good idea. There are some more great points made here.

If you have a look at the three implementations summarised here, you can see that there isn't a big difference between Dropout and DropConnect... it is just about scaling the values correctly, so that the (expected) sum remains consistent.

To each their own!

It almost gets philosphical! I believe it depends on your own personal perspective on why we use each of these layers. If you see Dropout as a means of restricting the network and preventing the co-adaptation of neurons, we are trying to block flow of information through certain paths of the network, so it might make sense to also remove that information from the batch-norm computations. If you see batch-norm as being a surgically precise method to man-handle the input distribution to a layer, you might want to compute batch-norm stats before the DropConnect sets any weights to zero and use those on the entire outbound batch.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.