Why normalize when all features are on the same scale?

Question

So I'm doing the tensorflow tutorial found here:

https://www.tensorflow.org/tutorials/keras/basic_classification

Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector.

The tutorial then says:

We scale these values to a range of 0 to 1 before feeding to the neural network model. For this, cast the datatype of the image components from an integer to a float, and divide by 255.

My question is why do we need to normalize in this case? My understanding is that when we have features that are on different scales, we need normalization if not the output of the model is distorted. But in this case all pixel ranges go from 0 to 255 (all features are the same scale)

I went ahead and ran it with normalization, and get an accuracy of over 85%, whereas no normalization, my accuracy falls to 10%.

Any ideas?

Answer 1

For neural networks, there is another reason. Sigmoid function provides values between 0 and 1; if the task is binary classification, you'd use a sigmoid function at the output. For another task, you might have used tanh function at some layer, a centered input to that neuron works computationally well.

The reason is as follows; outside the range of some input values, the derivatives of the activation functions will be close to zero. At those points, the gradient descent steps will be extremely slow due to the small weight updates. Directly think of the 2-D function graph of the related activation function and think of the range of input values where the derivatives are close to zero. This is why Rectified Linear Unit (ReLU) or Leaky Rectified Linear Unit (Leaky ReLU) outperforms the others in most of the tasks, with the constant derivative of 1, when the input is greater than 1.

Also refer to: https://stats.stackexchange.com/questions/51012/must-i-normalize-inputs-into-a-perceptron-that-uses-a-sigmoid-activation-functio

Answer 2

All of your features are on the same scale, true; however, there's no guarantee that they are evenly distributed along that scale. Depending on the image, you may have a large background of low-valued features and a small image of high-valued features, which will be swamped.

Normalization gives your features a mean of 0 and a standard deviation of one, which means that a feature with a mean of 223 won't dominate a feature with a mean of 10 or vice versa.