Suppose I want to use the Gradient Descent algorithm. I have a training set and a test set and I want to do the feature scaling with mean normalization.
Should I use the same mean and variance for equivalent features in the 2 sets? Why?
$\begingroup$
$\endgroup$
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
You do feature scaling for accelerating learning process. Features may have different scales. One maybe from 1 to 10 and one may be from -100 to 1000. Using normalization, you make the scale of them the same as each other, helps accelerate the learning process. You should find the mean and variance for each feature separately on your training data. then during training and testing each feature should be reduced by the corresponding mean and be divided by the corresponding standard deviation. So yes, for each feature during testing and training you have to provide same values for mean and std which are obtained using training data. I suggest you taking a look at here.
answered Jan 6, 2018 at 18:23
$\begingroup$
$\endgroup$
The main reason to use statistics computed on only the training set is to avoid leaking information from the test set.
If this is not a concern, then it is perfectly OK to use statistics from the entire data set.
See here for further discussion.
$\begingroup$
$\endgroup$
With scaling (or Z-transformation), you need a mean and a variance, which should come from total data.
What's more, if your model is going to be used on future coming data, then this mean and variance need to be applied to new data as well, as oppose to based on new data's mean or variance.
One important assumption in modelling is that feature / pattern in training would be the same or similar in testing set. This is the base that we can use history data to predict future.
Hence this requires consistency in mapping values to another using one transformation function. Different sets of mean and variance, would lead to different transformation.
$\begingroup$
$\endgroup$
The argument of avoiding information leaking into the new data by using training data statistics (mean, standard deviation) for normalization seems counter-intuitive because inserting training mean and standard deviation into the new data IS information leak from training data to new data.
Statistically, this procedure "forces" the membership of the new data to the training population which should not be necessarily true in all cases: prediction is also a test of stationarity of underlying processes.
For example, statistical t-tests for mean differences use each group's statistics therefore being able to detect (mean) differences between groups; this would be difficult (the test would have a lower power) if combined (or, only one of the group's) statistics would be used instead.
I think the current practice tries to keep the prediction error low by "bleeding" information from the training set into the new set.
$\begingroup$
$\endgroup$
Before answering or sharing my opinion I, will like to extend the question beyond just training and testing to include prediction, that too real time prediction and not batch mode prediction.
Recommended sequence of steps during modeling.
- Divide the sample data in training and validation set.
- Scale training data.
- Using same factor as training data (example mean and variance of training data) scale test data.
For in production prediction in real time use the above stored value to scale the feature. Update these values during model retraining. Refer here for detail discussion.
