0
$\begingroup$
  1. In Batch Normalization, mean and standard deviation are calculated feature wise and normalization step is done instance wise and in Layer Normalization mean and standard deviation are calculated instance wise nd normalization step is done feature wise, is this right or not?

  2. And in group normalization I understand that batches are made of group of features in the dataset? how are mean/standard deviation and normalization operation done here?

$\endgroup$

1 Answer 1

0
$\begingroup$
  1. Correct, In Batch Normalization, the mean and standard deviation are calculated feature-wise, and the normalization step is done instance-wise. For each feature, the mean and standard deviation are calculated across the instances in the batch, and then each instance is normalized using these feature-wise statistics. This is in contrast to Layer Normalization, where the mean and standard deviation are calculated instance-wise, and the normalization step is done feature-wise. This means that for each instance, the mean and standard deviation are calculated across the features, and then each feature is normalized using these instance-wise statistics

  2. In Group Normalization, the mean and standard deviation, as well as the normalization operation, are done by calculating the mean and standard deviation of a subgroup in the input tensor and then applying a scale and offset factor to normalize the activations of a single sample. This makes Group Normalization suitable for recurrent neural networks as well, as it normalizes the activations of a single sample rather than working on batches

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.