It is stated that for: Feature Normalization -
The test set must use identical scaling to the training set.
And the point is given that:
Do not scale the training and test sets using different scalars: this could lead to random skew in the data.
Could someone explain what that means?
Generally speaking, best practice is to use only the training set to figure out how to scale / normalize, then blindly apply the same transform to the test set.
For example, say you're going to normalize the data by removing the mean and dividing out the variance. If you use the whole dataset to figure out the feature mean and variance, you're using knowledge about the distribution of the test set to set the scale of the training set - 'leaking' information.
The right way to do this is to use only the training set to calculate the mean and variance, normalize the training set, and then at test time, use that same (training) mean and variance to normalize the test set.
As for the point in your question, imagine using the training mean and variance to scale the training set and test mean and variance to scale the test set. Then, for example, a single test example with a value of 1.0 in a particular feature would have a different original value than a training example with a value of 1.0 (because they were scaled differently), but would be treated identically by the model. This is where the bias would come from.
Hope this helps!
To answer this question, let us take three scenarios.
Scenario 1:
scaled_dataset = (dataset - dataset_mean) / dataset_std_deviation
train, test = split(scaled_dataset)
Scenario 2:
train, test = split(dataset)
scaled_train = (train - train_mean) / train_std_deviation
scaled_test = (test - test_mean) / test_std_deviation
Scenario 3:
scaled_train = (train - train_mean) / train_std_deviation
scaled_test = (test - train_mean) / train_std_deviation
The “correct” way is Scenario 3.
I agree, it may look a bit odd to use the training parameters and re-use them to scale the test dataset. (Note that in practice, if the dataset is sufficiently large, we wouldn’t notice any substantial difference between scenarios 1-3 because we assume that the samples have all been drawn from the same distribution.)
Again, why Scenario 3? The reason is that we want to pretend that the test data is “new, unseen data.” We use the test dataset to get a good estimate of how our model performs on any new data.
Now, in a real application, the new, unseen data could be just 1 data point that we want to classify. (How do we estimate mean and standard deviation if we have only 1 data point?) That’s an intuitive case to show why we need to keep and use the training data parameters for scaling the test set.
To recapitulate: If we standardize our training dataset, we need to keep the parameters (mean and standard deviation for each feature). Then, we’d use these parameters to transform our test data and any future data later on
Let me give a hands-on example of why this is important!
Let’s imagine we have a simple training set consisting of 3 samples with 1 feature column (let’s call the feature column “length in cm”):
sample1: 10 cm -> class 2
sample2: 20 cm -> class 2
sample3: 30 cm -> class 1
Given the data above, we compute the following parameters:
mean: 20
standard deviation: 8.2
If we use these parameters to standardize the same dataset, we get the following values:
sample1: -1.21 -> class 2
sample2: 0 -> class 2
sample3: 1.21 -> class 1
Now, let’s say our model has learned the following hypotheses: It classifies samples with a standardized length value < 0.6 as class 2 (class 1 otherwise). So far so good. Now, let’s imagine we have 3 new unlabeled data points that you want to classify.
sample4: 5 cm -> class ?
sample5: 6 cm -> class ?
sample6: 7 cm -> class ?
If we look at the unstandardized “length in cm” values in our training dataset, it is intuitive to say that all of these samples likely belong to class 2. However, if we standardize these by re-computing the standard deviation and mean from the new data, we would get similar values as before (i.e., properties of a standard normal distribution) in the training set and our classifier would (probably incorrectly) assign the “class 2” label to the samples 4 and 5.
sample4: -1.21 -> class 2
sample5: 0 -> class 2
sample6: 1.21 -> class 1
However, if we use the parameters from your training set standardization, we will get the following standardized values
sample4: -18.37
sample5: -17.15
sample6: -15.92
Note that these values are more negative than the value of sample1 in the original training set, which makes much more sense now!
What if the training set is further divided into validation, should the validation set be scaled with the training mean and std?
