I'm studying the gradient descent algorithm for single hidden layer neural networks. Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.

Why mathematically do the normalized features carry the same information of the initial features?


1 Answer 1


I assume by mean normalization, you mean scaling each feature by subtracting the mean and dividing by standard deviation:

$$ x_{\text{scaled}} = \frac{x - \bar{x}}{\sigma} $$

where $x$ is a feature.

Even though you are changing all of the values of $x$, they are each being scaled by the same amount - initially, a translation by a constant (subtracting the mean), and then scaling by a constant (dividing by standard deviation).

Here's a two-dimensional, randomly generated dataset generated with scikit-learn's make_blobs function (left) and the scaled version using the above equation for the $x$- and $y$-coordinates (right):

Original data and mean normalised data

The $x$- and $y$-values for each point have changed, but they are still all in the same place relative to each other. If you look closely, you can see that the structure of the data is identical, even though it has been scaled a bit, and you could go back to something that looks more like the original by simply 'zooming in' on the data. Because the structure of the data is the same, we say no information was lost.

Now consider a transformation where we only take the $x$-value, and set all $y$-values to $0$:

Original data and transformed by only keeping x-coordinate

The structure of the data has changed, and there is no way to return to the original data by scaling or stretching space uniformly, so we say information was lost here. This is an extreme example, but hopefully it illustrates the point.

One way to think about it is to think if it is more or less difficult for a classifier to distinguish between the classes after the transformation. In the first case, we can draw a line that perfectly separates the two clusters just as easily with the original data or the normalised data, but in the second case, there is no such line that separates the transformed data.

By the way, if you normalise each example rather than each feature (as asked in your comment), for this data, you end up with something that looks like this:

Data normalised by example rather than feature

where all points land on either $(-1,1)$ or $(1,-1)$. This makes sense, because normalisation makes the range of the values span from $-1$ to $1$. When there are only two dimensions, one of them has to become $-1$ and the other has to become $1$. Hopefully it's fairly obvious that information is lost here, and it's generally not a good idea to do this.

This is quite a hand-wavy explanation and doesn't really cover any actual information theory concepts, but hopefully it gives you some intuition for this. If you want to dive deeper into the mathematical side of things, have a look at the Wikipedia article for information theory.

  • $\begingroup$ How can you say that the vectors carry the same information ? They are different :( . ps: also, why do we normalize the features and not the dataset vectors ? ( = why do we normalize the rows of the dataset and not the columns ? ) $\endgroup$
    – Poiera
    Jan 5, 2018 at 8:13
  • $\begingroup$ I've updated my answer to be a lot more visual. Hopefully this makes sense to you! $\endgroup$ Jan 5, 2018 at 23:53
  • $\begingroup$ You are fantastic . So clear. The last questions : how can we mathematically describe that ( in the first case ) "structure of the data is the same" ? ps: i think in the second case,after the normalization the points land in the INTERVAL [-1,1], so there more points than just two. $\endgroup$
    – Poiera
    Jan 6, 2018 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.