Batch Normalization vs Other Normalization Techniques

Question

In the context of neural networks, I understand that batch normalization ensures that activation at each layer of the neural net does not 'blow-up' and cause a bias in the network. However, I don't understand why it would be used as opposed to other normalization techniques such as Cosine or Weight normalization, these achieve the same goal and don't seem to be any more computationally complex.

Could someone please explain to me the advantages and disadvantages of using batch normalization vs other normalization techniques, and which contexts batch norm would be most beneficial?

Answer 1

Cosine normalisation is result of the fact that we bound dot product and hence decrease the variance, when we use cosine similarity or centered cosine similarity instead of dot product in neural networks (which is quasi ground-stone in NN)

Main benefit of cosine normalisation is Cosine normalization bounds the pre-activation of neuron within a narrower range, thus makes lower variance of neurons.

Also it does not depend on any statistics on batch or mini-batch examples, and performs the same computation in forward propagation at training and inference times. In convolutional networks, it normalizes the neurons from the receptive fields rather than the same layer or batch size.

Have a look at this paper showing emipirically comparison between normalisations you mentioned. C.N. comes on top.