1
$\begingroup$

Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because then the validation has some insight into the training data?

I'm not sure if I should either

1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

Many thanks

$\endgroup$
  • $\begingroup$ Just found the answer on stackoverflow (seems the answer is 1): stackoverflow.com/questions/49444262/… $\endgroup$ – BigBadMe Sep 26 '18 at 14:26
  • $\begingroup$ I believe a better idea would be to leave validation data completely untouched. $\endgroup$ – Random Nerd Nov 27 '18 at 5:51
1
$\begingroup$

Yes, the test data might contain something important that you didn't learn from the test set. No, you're not allowed to use the test set for learning.

The purpose of the test set is to understand how the system would behave on new data not seen during training. Fitting your scaler in the test set would defeat that purpose.

1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

That's what you should do.

Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

This is valid, if your production system makes it like that. So if you get data in batches and normalize the batches it might be fine as well.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.