4
$\begingroup$

Suppose I want to use the Gradient Descent algorithm. I have a training set and a test set and I want to do the feature scaling with mean normalization.

Should I use the same mean and variance for equivalent features in the 2 sets? Why?

$\endgroup$
1
$\begingroup$

You do feature scaling for accelerating learning process. Features may have different scales. One maybe from 1 to 10 and one may be from -100 to 1000. Using normalization, you make the scale of them the same as each other, helps accelerate the learning process. You should find the mean and variance for each feature separately on your training data. then during training and testing each feature should be reduced by the corresponding mean and be divided by the corresponding standard deviation. So yes, for each feature during testing and training you have to provide same values for mean and std which are obtained using training data. I suggest you taking a look at here.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The main reason to use statistics computed on only the training set is to avoid leaking information from the test set.

If this is not a concern, then it is perfectly OK to use statistics from the entire data set.

See here for further discussion.

| improve this answer | |
$\endgroup$
0
$\begingroup$

With scaling (or Z-transformation), you need a mean and a variance, which should come from total data.

What's more, if your model is going to be used on future coming data, then this mean and variance need to be applied to new data as well, as oppose to based on new data's mean or variance.

One important assumption in modelling is that feature / pattern in training would be the same or similar in testing set. This is the base that we can use history data to predict future.

Hence this requires consistency in mapping values to another using one transformation function. Different sets of mean and variance, would lead to different transformation.

| improve this answer | |
$\endgroup$
0
$\begingroup$

The argument of avoiding information leaking into the new data by using training data statistics (mean, standard deviation) for normalization seems counter-intuitive because inserting training mean and standard deviation into the new data IS information leak from training data to new data.

Statistically, this procedure "forces" the membership of the new data to the training population which should not be necessarily true in all cases: prediction is also a test of stationarity of underlying processes.

For example, statistical t-tests for mean differences use each group's statistics therefore being able to detect (mean) differences between groups; this would be difficult (the test would have a lower power) if combined (or, only one of the group's) statistics would be used instead.

I think the current practice tries to keep the prediction error low by "bleeding" information from the training set into the new set.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.