Should input images be normalized to -1 to 1 or 0 to 1

Question

Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming activation function is ReLu.

My question is, would normalizing images to [-1, 1] range be unfair to input pixels in negative range since through ReLu, output would be 0. Would normalizing images into [0, 1] range instead be a better idea?

Thank you.

Answer 1

In addition to just initialization (as the great answer of Djib2011 notes), many analyses of artificial neural networks utilize or rely on the normalization of inputs and outputs (e.g., the SELU activation). So normalizing the input is a good idea.

Often, however, this can be done with normalization layers (e.g., LayerNorm or BatchNorm), and furthermore, we may want to enforce that the pixels are in a particular fixed range (since real images are like this). This is especially important when the output is an image (e.g., for a VAE of images). Since we need to compare the input image $I$ to the output image $\widehat{I}$, it should be readily possible to enforce the pixel values of $\widehat{I}$ into a simple, known, hard range. Using sigmoid produces values in $[0,1]$, while using tanh does so in $[-1,1]$. However, it is often thought that that tanh is better than sigmoid; e.g.,

• https://stats.stackexchange.com/questions/142348/tanh-vs-sigmoid-in-neural-net

• https://stats.stackexchange.com/questions/330559/why-is-tanh-almost-always-better-than-sigmoid-as-an-activation-function/369538

• https://stats.stackexchange.com/questions/101560/tanh-activation-function-vs-sigmoid-activation-function

In other words, for cases where the output must match the input, using $[-1,1]$ may be a better choice. Furthermore, though not "standardized", the range $[-1,1]$ is still zero-centered (unlike $[0,1]$), which is easier for the network to learn to standardize (though I suspect this matters only rather early in training).

Also, for this phrase

would normalizing images to [-1, 1] range be unfair to input pixels in negative range since through ReLu, output would be 0

the answer is "no". Mainly because the non-linear activation happens after other layers first in nearly all cases. Usually those layers (e.g., fully connected or conv) have a bias term which can and will shift around the range anyway (after some additional, usually linear, transformation occurs).

It is true however that values below zero do "die" wrt to their contribution to the gradient. Again, this may be especially true early in training. This is one argument for using activations other than ReLU, like leaky ReLU, and it is a real danger. However, the hope is that these values should have more than one way to propagate down the network. E.g., multiple outputs in the first feature map (after the first convolutional layer, before the activation) will depend on a given single input, so even if some are killed by the ReLU, others will propagate the value onwards. This is thought to be one reason why ResNet is so effective: even if values die to ReLU, there is still the skip connection for them to propagate through.

Despite all this, it is still probably more common to normalize images with respect to the statistics of the whole dataset. One problem with per-image normalization is that images with very small pixel value ranges will be "expanded" in range (e.g., an all blue sky with a tiny cloud will immensely highlight that cloud). Yet, others may consider this a benefit in some cases (e.g., it may remove differences in brightness automatically).

Ultimately, the optimal approach is up for debate and likely depends on the problem, data, and model. For more, see e.g. [1], [2], [3], [4], [5], [6], [7]

Answer 2

As the other answers previously said, in practice it doesn't have much difference which of the two you choose. However, theoretically it's better to scale your input to $[-1, 1]$ than $[0, 1]$ and I'd argue that it's even better to standardize your input (i.e. $μ=0$, $σ=1$).

Let me explain why:

Deep neural networks, especially in their early days, had much trouble with backpropagation, as they suffered from vanishing/exploding gradients. A popular way to combat this is by initializing the weights of the network in smarter ways. Two initialization techniques have proven to be the most popular: Glorot initialization (sometimes referred to as Xavier initialization) and He initialization, both of which are variations of the same idea.

In order to derive their intialization they make a few assumptions, one of which is that the input features have a zero mean. You can read this post for simplified explanation, but I'd suggest reading the two papers mentioned above.

Answer 3

Its better if you we consider normalization 0 to 1, there are 2 reasons.

1. if we check with the train data, if there is pixel then it will be some value and empty value will be 0, hope we don't see any -1.
2. If we are using activation function "rectified linear unit(relu)," it will convert output of a neuron is always greater than 0 nothing like negative values. If you are using other activation function like "tanh" or "sigmod" then there might be a chance for negative values.

For better results use activation functions wisely.

Answer 4

I think it doesn't matter much.

Firstly because network doesn't know that input is pixel value, which can't be negative.

Secondly even though relu will cut the negative values, the network will adjust its weight to compensate for that.