7
$\begingroup$

In Let's build GPT: from scratch, in code, spelled out., Andrej Karpathy says that no one likes Batch Normalization layer and people want to remove it. He also said it brings so many bugs and he shot his foot by this.

But I am not sure why Batch Normalization is so undesirable. Kindly give concrete examples and mechanism that cause the harms for each situation.

Improve this question
$\endgroup$
1
  • $\begingroup$ @AlbertoSinigaglia, please click the link. $\endgroup$
    – mon
    Commented Jan 26 at 13:29

1 Answer 1

Reset to default
7
$\begingroup$

Problems of batch normalization:

  • Unstability with small batch size: with only a few samples on the batch, the presence of noise/outliers can pull the statistics away from the population values.
  • Unstable training dynamics in general (source).
  • Longer training times (source).
  • Different behavior between training and inference times, making it more difficult to reason when finding unexpected values in the training and validation losses.
  • Doesn't make sense in RNN architectures, where weights are shared for all timesteps, but the output of each one might have different statistics. Recurrent Batch Normalization was proposed for that.
  • Coupling in the computation of different training samples in the minibatch. Can't think of an scenario where is was actually a problem, but it's something to consider.
Improve this answer
$\endgroup$
1
  • $\begingroup$ @mon did the answer solve your doubts? $\endgroup$
    – noe
    Commented Mar 23 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.