In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous layer's node values. Let's say we have a network like this (taken from the video):
As we change the mini-batch that the neural network is trained on, it's possible that the distribution of the input features with respect to different mini-batches is different. As far as I understand, this is called covariate shift. An issue with covariate shift is that maybe the distribution of input features with respect to the validation set examples is different from the training set (ref. black cats / colored cats example in the linked video), and the distributions w.r.t. the different mini-batches are different.
Consequently, since the learned weights connecting the input layer to the next layer depend on the input features distribution, the learned weights will accordingly change with each mini-batch. Thus the activations in the next hidden layer will also vary in their distribution w.r.t. different mini-batches.
First question: Is my understanding so far correct?
Now I can understand that limiting the mean and variance of input features or that of the hidden layer activations will address the covariate shift problem only if the distributions across mini-batches (or between training and validation sets) are fundamentally the same but differ in parameters (not the neural network parameters - I mean mean and stddev). Crudely speaking, if the distribution of input features w.r.t different mini-batches is normal, but the parameters (mean and stddev) are different, then batch normalization seems fine.
Question 2: But what if the input features have normal distribution w.r.t. one mini-batch and a completely different distribution w.r.t. another? I don't see how limiting the mean and variance will help. Maybe I'm ignoring some key assumptions.
Question 3: Why does limiting the variation in the distributions of the hidden layers speed up the convergence?