Should we always use Batch Renormalization over Batch Normalization?

Question

According to a machine level mastery post on batch norm

For small mini-batch sizes or mini-batches that do not contain a representative distribution of examples from the training dataset, the differences in the standardized inputs between training and inference (using the model after training) can result in noticeable differences in performance. This can be addressed with a modification of the method called Batch Renormalization (or BatchRenorm for short) that makes the estimates of the variable mean and standard deviation more stable across mini-batches.

My question is: Can we just always use Batch Renormalization then? In what cases, would Batch Normalization be better than Batch Renormalization?

Answer 1

The main drawback of batch renormalization is that there are extra hyperparameters to tune. If your data meets the conditions where batch normalization works, then you will get no advantage from batch renormalization. In the example used in Figure 1 of Ioffe's paper Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models, the batch normalisation/renormalisation results are almost identical for a batch size of 32, so it appears that at least in some cases, you don't need a very large batch size before the benefit of batch renormalisation disappears. At this point, any effort/computational resources you spend tuning the batch renormalization hyperparameters are wasted.

Where batch renormalization does make a difference it's still a trade-off. In Timon Ruban blog A Refresher on Batch (Re-)Normalization he says "As most things in life it is a trade-off between your time and your model’s performance. If you have outsourced your hyperparameter tuning to things like Bayesian optimization, it’s at least still a trade-off between computing resources and performance."

Answer 2

Here some describe some points below :

Smaller datasets: Batch Renormalization relies on computing the mean and standard deviation estimates across mini-batches. For smaller datasets, this can lead to unreliable estimates of the mean and standard deviation. In such cases, Batch Normalization may be a better choice.

When training time is a priority: Batch Renormalization introduces additional computations, which can increase training time. If training time is a priority, Batch Normalization may be a better choice as it is computationally less expensive.

Non-stationary datasets: Batch Renormalization assumes that the statistical distribution of input features across mini-batches is stationary. However, in some cases, such as in online learning or datasets with significant concept drift, the statistical distribution may change over time. In such cases, Batch Normalization may be more appropriate.

Models with gated activations: Batch Renormalization may not be well-suited for models with gated activations such as LSTM or GRU. This is because the gating mechanism can interact with the Batch Renormalization layer and affect the estimates of mean and variance.

while Batch Renormalization can improve the stability of the mean and standard deviation estimates across mini-batches, it is not always the best choice. The choice between Batch Normalization and Batch Renormalization depends on the characteristics of the dataset and the model architecture.