2
$\begingroup$

I'm using Scikit Learn to guess the tag of Stack Overflow posts given the title and body. I represent the title and body as two 300-dimensional vectors of floats. The documentation for Scikit learn's SGDClassifier states:

For best results using the default learning rate schedule, the data should have zero mean and unit variance.

With one-dimensional data, I understand what mean and variance are but when we go into higher dimensions how would one calculate the mean and variance for a dataset?

| improve this question | | | | |
$\endgroup$
2
$\begingroup$

It's just the expansion of one dimensional mean and standard deviation. Suppose that you are trying to estimate the weight of a person and you have two inputs, salary and height. For finding the weight, you have two inputs which are of different scales, so you try to find the mean and variance of each feature, salary and height, separately using the data samples. Suppose you have two data samples in a tuple like (salary, height). They are as follows:

(10000, 180)
(5000, 175)

Although the first guy may seem so rich, the real point about the first and second data samples is that their scale is not the same. The approach is like this:

  1. Find the mean of first feature among data samples, salary here. (10000 + 5000) / 2 = 7500
  2. Find the mean of second feature among data samples, height here. (180 + 175) / 2 = 177.5
  3. Find the standard deviation of first feature, among data samples, which is 2500 for salary.
  4. Find the standard deviation of second feature, among data samples, which is 2.5 for height.
  5. Reduce the amount of mean from corresponding feature for each data sample and divide each with the standard deviation of corresponding feature.
    (10000 - 7500) / 2500 = 1
    (5000 - 7500) / 2500 = -1

    (180 - 177.5) / 2.5 = 1
    (175 - 177.5) / 2.5 = -1

Data samples which have been normalized now are like as follows:

(1, 1)
(-1, -1)

Whenever you normalize your data, your cost function would be so easier to learn. The weights don't have to struggle to reach to high values in situations where your features are not in same range.

| improve this answer | | | | |
$\endgroup$
  • 1
    $\begingroup$ That makes a lot of sense. Thank you for the explanation. Very clear! $\endgroup$ – Aidan Rosswood Jan 28 '18 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.