Let me present you with a toy example and a reasoning on image normalisation I had:
Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (black) to 255 (white).
- Class 0: Images that contain a filled circle in the centre of the image. The pixel values within the circle are high, close to 255. Pixels outside the circle are low, close to 0. Below is an example of a class 0 picture.
- Class 1: Images that are empty, containing pixels close to 0. An example is below, basically a dark image.
Note: The images show basically binary values: white/black, however, assume that black values could be in the range of [0, 30] and white ones [230-255] with some random distribution.
Prior to feeding the network with the images, we pre-process and normalize the data. This basically means (i) centring the data and (ii) scaling the data. A typical and simple approach could be to subtract 127.5 to all pixels and scale by 255 so as to have pixel values ranging [-1, 1]. Alternatively, another approach consists of computing the "mean image" and the "dynamic-range image" and centre and scale the images pixel-wise. Let us focus on the latter.
Computing the "mean image" mainly means summing all images pixel-wise and taking the average pixel-wise. As a result, we obtain a NxN average image. This can sometimes shed some light on our task and help us understand the data we are working with. Likewise, the "dynamic range image" is obtained by computing the maximum and minimum values pixel-wise. The "dynamic range image" shows us which range of values each pixel can take.
During training, the kernels of the different layers in the CNN will be trying to detect the circle. Since the task is rather simple, the kernels basically would have to learn that there is a strong pixel-value-transition from "empty region" to "circle region" when a circle is present.
After normalising the images pixel-wise, we have that all pixels have the same range and same mean, namely [-1, 1] and 0, respectively.
Consider a class 0 image (i.e. image has a circle). Before normalizing, pixel (i, j) has value 30 (dark), which happens to be the maximum that pixel (i, j) ever takes. Pixel (i, j+1) has a value 255 (white), also the maximum value that pixel (i, j+1) ever takes. So we have a strong transition that a kernel from first layers should be able to learn.
But, if we normalise the data with "mean image" and "dynamic range image" isn't this strong transition going to be softened? In the particular, the transition would be from value 1 to value 1, since both were the maximums at their respective positions. From a clear and hard transition we have ended with a transitioned from similarly valued pixels.
Conclusion Using this pixel-wise normalization we end up losing global pixel information, which might make the job of the kernels more difficult. I believe same conclusions might be drawn if "variance image" is used instead of "dynamic range image".
Question Am I missing some point or is this normalisation no longer used?
Bonus I also thought of Batch Normalisation, which sometimes is used at the first layer so that normalisation is done within the network. I thought that maybe the parameters "beta" and "gamma" used in BN might be able to fix this issue if it ever appears while training.