1
$\begingroup$

So I'm doing the tensorflow tutorial found here:

https://www.tensorflow.org/tutorials/keras/basic_classification

Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector.

The tutorial then says:

We scale these values to a range of 0 to 1 before feeding to the neural network model. For this, cast the datatype of the image components from an integer to a float, and divide by 255.

My question is why do we need to normalize in this case? My understanding is that when we have features that are on different scales, we need normalization if not the output of the model is distorted. But in this case all pixel ranges go from 0 to 255 (all features are the same scale)

I went ahead and ran it with normalization, and get an accuracy of over 85%, whereas no normalization, my accuracy falls to 10%.

Any ideas?

$\endgroup$
  • $\begingroup$ So there are lots of parameters that depend on input range. Eg initialisation weights, weight decay..so you are kind of right that it should not matter, but you then have to change the parameters too.. $\endgroup$ – seanv507 Oct 19 '18 at 19:26
1
$\begingroup$

For neural networks, there is another reason. Sigmoid function provides values between 0 and 1; if the task is binary classification, you'd use a sigmoid function at the output. For another task, you might have used tanh function at some layer, a centered input to that neuron works computationally well.

The reason is as follows; outside the range of some input values, the derivatives of the activation functions will be close to zero. At those points, the gradient descent steps will be extremely slow due to the small weight updates. Directly think of the 2-D function graph of the related activation function and think of the range of input values where the derivatives are close to zero. This is why Rectified Linear Unit (ReLU) or Leaky Rectified Linear Unit (Leaky ReLU) outperforms the others in most of the tasks, with the constant derivative of 1, when the input is greater than 1.

Also refer to: https://stats.stackexchange.com/questions/51012/must-i-normalize-inputs-into-a-perceptron-that-uses-a-sigmoid-activation-functio

$\endgroup$
0
$\begingroup$

All of your features are on the same scale, true; however, there's no guarantee that they are evenly distributed along that scale. Depending on the image, you may have a large background of low-valued features and a small image of high-valued features, which will be swamped.

Normalization gives your features a mean of 0 and a standard deviation of one, which means that a feature with a mean of 223 won't dominate a feature with a mean of 10 or vice versa.

$\endgroup$
  • 1
    $\begingroup$ For standard normalization this may be the case, but for the "normalization" being used here (which is simply dividing by 255 to get a number in the range 0-1), that's not a factor. $\endgroup$ – R.M. Nov 19 '18 at 21:26
  • $\begingroup$ Makes a difference, I agree. Suggestion? For terms like "normalization" that have a fairly standard definition, it will clarify things to either find a different standard descriptor or describe, as you just did, what you intend. Would help me, at least, and I need all the help I can get to be helpful! $\endgroup$ – Richard Careaga Nov 20 '18 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.