2
$\begingroup$

Question

For CNN to recognize images, why not use the entire batch data, instead of per feature, to calculate the mean in the Batch Normalization?

When each feature is independent, need to use per feature. However the features (pixels) of images having RGB channels with 8 bit color for CNN are related. If there are 256 pixels in R channel in an image, 255 for pixel i and 255 for pixel j are both white meaning the same intensity(?) in R color.

Then why not use the mean of the entire data in a batch? If the pixel channel i happens to have the values between (0, 127) and channel j has (128, 255), the meaning that (0, 127) is within [0, 255] and the relational meaning between i and j, which is, pixel i intensity is lower than that of j) gets lost.

enter image description here

Improve this question
$\endgroup$
1
  • $\begingroup$ Layer Normalization does normalize all channels together, so it's certainly a reasonable thing to do. So the answer is: there isn't necessarily a deep reason for why BN does it like this. Deep learning is quite empirical and it was even more so in 2015 when BN was introduced. $\endgroup$
    – isarandi
    Commented Nov 19, 2023 at 14:23

1 Answer 1

Reset to default
2
$\begingroup$

The mentioned case is actually correct if you apply BN for an input layer. 😌

However, the point is that BN is used mostly after convolutional or fully-connected layers (before an activation layer). Therefore, BN will be calculated not for pixel values but for the outputs of conv or F-C layers, which are non-limited in terms of range (a value could be anything from -inf to +inf). 🤸‍♂️

And moreover, outputs of conv/F-C layer will have completely different dimensions (since you applied different kernels), and therefore, can't be actually treated as "somehow related" to each other.

Edited: added fully-connected layers

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the input. So the filter in the conv layer, condense in the pool layer, non-linear (-ish) transformation at activation layer break the co-relations among channels? $\endgroup$
    – mon
    Commented Sep 16, 2021 at 0:07
  • $\begingroup$ I slightly improved my description and specified the layer positions, I think now it's more clear. Yes, the correlations are no more right after your input values came through the first convolutional layer. I mean, of course, you still have spatial information preserved, however in convolutional layers the dimensions (channels) are completely different from the dimensions (channels) you had in the input layer. $\endgroup$
    – Dmitry Demidov
    Commented Sep 16, 2021 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.