1
$\begingroup$

Data normalization: It ensures that each input (each pixel value, in this case) comes from a standard distribution. This standardization makes our model train and reach a minimum error, faster!

my question is about standardization, I don't understand the relation between input data being between 0 and 1 to gradient decent being faster or even being correct(predicts well), why is that, I'm very ok with a deep math proof.

$\endgroup$

1 Answer 1

1
$\begingroup$

The standardization of the pixel values comes from the same logic standardization data to make Gradient Descent (or some other similar optimization algorithm) faster. When you provide an image to the NN, it is no different from other data types, meaning you provide a sample with different features (or let's say variables). When an image is a case, your features are your pixels. That is, if you have a 32x32 image this means you have 1024 features. If your input is RGB, it creates 3 dimensions which make your feature size 32x32x3 that is, 3072. In other words, your variable in such a case is a specific pixel of all the images in your dataset. Your image is just a single row of data with 3072 features like in a simple regression problem where for example, house price is predicted using 5 features location, size, age of the building, number of rooms, and floor.

Now, before going into details, let's first think why distribution might be different. Depending on your image type, your specific channel values may have a bigger variance than others. For example, if your data samples, images, are about plants, trees, or forest then your Green and Blue channel values will probably vary a lot, but your Red channel will always have a value close to 0. Thus, the features that represent these channels will have a different distribution than others.

Another thing that needs to be considered is pixels' (even if it is a grayscale image) having dissimilar distribution due to natural reasons. For example, consider the dataset where all images are identity card photos of people, where the background is white. In such a case, some of the pixels, probably the top left and top right ones will have a variance close to zero due to having almost the same color value for all images.

These two cases cause the possibility of a specific pixel having, for example, values 0-10, another one having 3-240, another one have 126-255, etc. In such a case, what needs to be handled is the same problem as Andrew NG talks about in this video. Since the ranges of your values will vary in 3072 (for example) dimensions, gradient descent will need more steps to find optimal value in one dimension when compared to the other dimension. This is because your step size is fixed. Consider, you have 100 rungs in one ladder and 12 rungs in another ladder. With normal human step size 1, which one would be faster to climb? The latter.

To briefly summarize, since for all features in the NN same parameters are used, to expect those parameters affect the features in the same way (or objectively), these features should display similar characteristics. If not, some features might advantage more from these parameters when compared to others.

$\endgroup$
2
  • 1
    $\begingroup$ very clear, thank you very much $\endgroup$ Nov 5, 2020 at 11:02
  • 1
    $\begingroup$ Dear Abdulrahman I suggest you to upvote the answer if you think it is useful :) $\endgroup$
    – German C M
    Nov 5, 2020 at 12:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.