What do mean and variance mean for high dimensional data?

Question

I'm using Scikit Learn to guess the tag of Stack Overflow posts given the title and body. I represent the title and body as two 300-dimensional vectors of floats. The documentation for Scikit learn's SGDClassifier states:

For best results using the default learning rate schedule, the data should have zero mean and unit variance.

With one-dimensional data, I understand what mean and variance are but when we go into higher dimensions how would one calculate the mean and variance for a dataset?

Answer 1

It's just the expansion of one dimensional mean and standard deviation. Suppose that you are trying to estimate the weight of a person and you have two inputs, salary and height. For finding the weight, you have two inputs which are of different scales, so you try to find the mean and variance of each feature, salary and height, separately using the data samples. Suppose you have two data samples in a tuple like (salary, height). They are as follows:

(10000, 180)
(5000, 175)

Although the first guy may seem so rich, the real point about the first and second data samples is that their scale is not the same. The approach is like this:

1. Find the mean of first feature among data samples, salary here. (10000 + 5000) / 2 = 7500
2. Find the mean of second feature among data samples, height here. (180 + 175) / 2 = 177.5
3. Find the standard deviation of first feature, among data samples, which is 2500 for salary.
4. Find the standard deviation of second feature, among data samples, which is 2.5 for height.

5. Reduce the amount of mean from corresponding feature for each data sample and divide each with the standard deviation of corresponding feature.
   (10000 - 7500) / 2500 = 1
(5000 - 7500) / 2500 = -1
   

   (180 - 177.5) / 2.5 = 1
(175 - 177.5) / 2.5 = -1
   

Data samples which have been normalized now are like as follows:

(1, 1)
(-1, -1)

Whenever you normalize your data, your cost function would be so easier to learn. The weights don't have to struggle to reach to high values in situations where your features are not in same range.