In the tutorial, they normalize the data and say "The mean and standard deviation should only be computed using the training data"

What does this refer to? Why should you only use the training data?


When building any Machine Learning model, the only observable data you have is training data. Test data is supposed to be unobserved data, meaning that even though you might have it now, you need to act as if you didn't. When you apply normalisation, you first observe the data to get the parameters you need. As you are only supposed to be able to observe the training data, you can't use the test data to calculate those values. Doing so would be like cheating, as you are accomodating your parameters to new unobserved data (how can you observe unobserved data?).

Imagine you build a model today and you want to make predictions tomorrow. You can't use tomorrow data to build your model since you don't have it yet. You are not supposed to know tomorrow's mean and std, though your hope is that they will be similar enough. That is why when you normalose/standarise you get the parameters with the training data and then use them to transform both train and test data, so you can use them as inputs for your model.

  • $\begingroup$ Great explanation, thank you. I notice the tf.keras.utils.normalize function doesn't seem to allow you to supply the length of the training data during normalization. How does that function expect you to be able to normalize the data with only the training std and mean if it doesn't know how to separate them? $\endgroup$ – raeldor May 12 '20 at 12:56
  • $\begingroup$ As far as I know, tf.keras.utils.normalize does not use mean and std (nor min max scaler). It uses Lp-norms (check this question. $\endgroup$ – TitoOrt May 12 '20 at 13:07
  • $\begingroup$ As with any normalization method you must feed it just the data you want it to see. If your validation_split is 0.1 then you'd use the first 90% of the data to estimate normalization parameters. $\endgroup$ – NikoNyrh May 12 '20 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.