Data preprocessing: Should we normalise images pixel-wise?

Question

Let me present you with a toy example and a reasoning on image normalisation I had:

Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (black) to 255 (white).

• Class 0: Images that contain a filled circle in the centre of the image. The pixel values within the circle are high, close to 255. Pixels outside the circle are low, close to 0. Below is an example of a class 0 picture.

• Class 1: Images that are empty, containing pixels close to 0. An example is below, basically a dark image.

Note: The images show basically binary values: white/black, however, assume that black values could be in the range of [0, 30] and white ones [230-255] with some random distribution.

Prior to feeding the network with the images, we pre-process and normalize the data. This basically means (i) centring the data and (ii) scaling the data. A typical and simple approach could be to subtract 127.5 to all pixels and scale by 255 so as to have pixel values ranging [-1, 1]. Alternatively, another approach consists of computing the "mean image" and the "dynamic-range image" and centre and scale the images pixel-wise. Let us focus on the latter.

Computing the "mean image" mainly means summing all images pixel-wise and taking the average pixel-wise. As a result, we obtain a NxN average image. This can sometimes shed some light on our task and help us understand the data we are working with. Likewise, the "dynamic range image" is obtained by computing the maximum and minimum values pixel-wise. The "dynamic range image" shows us which range of values each pixel can take.

Hypothesis 1

During training, the kernels of the different layers in the CNN will be trying to detect the circle. Since the task is rather simple, the kernels basically would have to learn that there is a strong pixel-value-transition from "empty region" to "circle region" when a circle is present.

Hypothesis 2

After normalising the images pixel-wise, we have that all pixels have the same range and same mean, namely [-1, 1] and 0, respectively.

Final Hypothesis

Consider a class 0 image (i.e. image has a circle). Before normalizing, pixel (i, j) has value 30 (dark), which happens to be the maximum that pixel (i, j) ever takes. Pixel (i, j+1) has a value 255 (white), also the maximum value that pixel (i, j+1) ever takes. So we have a strong transition that a kernel from first layers should be able to learn.

But, if we normalise the data with "mean image" and "dynamic range image" isn't this strong transition going to be softened? In the particular, the transition would be from value 1 to value 1, since both were the maximums at their respective positions. From a clear and hard transition we have ended with a transitioned from similarly valued pixels.

Conclusion Using this pixel-wise normalization we end up losing global pixel information, which might make the job of the kernels more difficult. I believe same conclusions might be drawn if "variance image" is used instead of "dynamic range image".

Question Am I missing some point or is this normalisation no longer used?

Bonus I also thought of Batch Normalisation, which sometimes is used at the first layer so that normalisation is done within the network. I thought that maybe the parameters "beta" and "gamma" used in BN might be able to fix this issue if it ever appears while training.

Answer 1

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes:

$ z = \frac{x - \mu}{\sigma} $

where:

$ x $ is the pixel value

$ \mu $ is the arithmetic mean of the channel distribution

$ \sigma $ is the standard deviation of the channel, calculated as such:

$ \sigma = \sqrt{\frac{\sum^N_{i=1} |x_i-\mu|^2}{N}} $

By standardizing the data in this fashion, we obtain a mean ($\mu$) = 0 and a standard deviation ($\sigma$) = 1. This allows SGD to converge faster.

It is easier to see how feature scaling can benefit simpler models such as SVMs, but CNNs benefit similarly. (Mini-)Batch normalization makes every pixel have a similar distribution, which helps SGD converge faster. This is because networks generally have shared hyperparameters. For instance, the learning rate is a multiplier for all weights in a network, rather than having a learning rate for every weight; therefore, we want a single learning rate to have proportional influence in all regions. Otherwise, the learning rate could make some weights over-correct and other weights under-correct.

---

This article does a great job of explaining CNN preprocessing. I'll try to highlight the key points (the following images are pulled from it)

Let's consider a face recognition challenge:

Taking the mean (left) and standard deviation (right) of the batch, we get the following:

Our centered data resembles a face, but more importantly our standard deviation image shows great variance along the borders and everywhere except the face. From our z-score formula, we can predict that the standardized values near the border and irrelevant areas will be relatively squashed more than values around the face. The resultant normalization appears as such:

---

Normalizing the input images is widely used and often considered an essential preprocessing step. However, making an absolute statement such as, "normalization is always beneficial" or "normalization is never beneficial" would be misguided.

In the scenario you presented, normalizing the data would not benefit the CNN, but the network would still be able to learn the data well. However, this is a very primitive situation, and most real-world problems have significantly more noise and variance for which to account. In general, normalizing the input image will improve SGD convergence time.