What I understand so far...

The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift".

Covariance shift is a difference in distribution of features between the training dataset and inference dataset. But this distribution of features actually is a metadistribution: if you view a single data point as a distribution of features, then the covariance shift is describing the difference in distribution of distribution of features. The fact that instance normalization worked makes me think that this view is more accurate, because what instance normalizatoin does is normalize the distribution of features within a single instance, not amongst all other instances within a batch (which is what BatchNorm does).


The potentially pedantic description of describing distribution of features as meta distribution was leading up to this question: it makes intuitive sense that normalizing features of all instances within a single batch can help covariance shift, since it is directly normalizing the meta-distribution. Then, what is it the implication of normalizing every individual distribution within the meta-distribution (i.e. InstNorm) in terms of covariance shift? I simply don't know enough about stats to predict the behavior of distribution of distribution, when each distribution within the meta-distribution is normalized.

I understand that my question isn't really well formed and explained, so I will fix it if you point out the obscurities.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.