I'm studying the gradient descent algorithm for single hidden layer neural networks. Suppose that I have an initial dataset and then I use mean normalization in order to scale the features.
Why mathematically do the normalized features carry the same information of the initial features?
I assume by mean normalization, you mean scaling each feature by subtracting the mean and dividing by standard deviation:
$$ x_{\text{scaled}} = \frac{x - \bar{x}}{\sigma} $$
where $x$ is a feature.
Even though you are changing all of the values of $x$, they are each being scaled by the same amount - initially, a translation by a constant (subtracting the mean), and then scaling by a constant (dividing by standard deviation).
Here's a two-dimensional, randomly generated dataset generated with scikit-learn's make_blobs function (left) and the scaled version using the above equation for the $x$- and $y$-coordinates (right):
The $x$- and $y$-values for each point have changed, but they are still all in the same place relative to each other. If you look closely, you can see that the structure of the data is identical, even though it has been scaled a bit, and you could go back to something that looks more like the original by simply 'zooming in' on the data. Because the structure of the data is the same, we say no information was lost.
Now consider a transformation where we only take the $x$-value, and set all $y$-values to $0$:
The structure of the data has changed, and there is no way to return to the original data by scaling or stretching space uniformly, so we say information was lost here. This is an extreme example, but hopefully it illustrates the point.
One way to think about it is to think if it is more or less difficult for a classifier to distinguish between the classes after the transformation. In the first case, we can draw a line that perfectly separates the two clusters just as easily with the original data or the normalised data, but in the second case, there is no such line that separates the transformed data.
By the way, if you normalise each example rather than each feature (as asked in your comment), for this data, you end up with something that looks like this:
where all points land on either $(-1,1)$ or $(1,-1)$. This makes sense, because normalisation makes the range of the values span from $-1$ to $1$. When there are only two dimensions, one of them has to become $-1$ and the other has to become $1$. Hopefully it's fairly obvious that information is lost here, and it's generally not a good idea to do this.
This is quite a hand-wavy explanation and doesn't really cover any actual information theory concepts, but hopefully it gives you some intuition for this. If you want to dive deeper into the mathematical side of things, have a look at the Wikipedia article for information theory.
