3
$\begingroup$

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, we apply transform() testing data so that the learned parameters from scaling the train data are applied on testing data. Similar to this:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

The question is: Does it also make sense to perform fit_transform() on the training data but NOT transform() testing data at all so that we get to test the model performance on actual real-world data that are not transformed at all?

Thank you

$\endgroup$
5
$\begingroup$

No, it does not make sense to do this.

You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen data.

By not performing the same scaling on the test data, you are introducing systematic errors in the model. This was pointed out in the comments by nanoman - see that comment for a simple transformation example.

To exaggerate the point, imagine you have a language model, translating text from English to French. You apply a single transformation on your English data: you first translate it to Spanish. Now the model is trained to translate Spanish to French, with some accuracy. Now you move on to the test data - still in English - and you do not apply the same transformation as you did to your training data. You are asking the model to translate directly from English to French, instead of Spanish to French, and it is obvious that the results won't be good.

In principal, this idea is the same as with any other model and transformations, just that the impact might not always be so visible i.e. you might get really lucky and not notice a large impact.

The language model might have learnt some elementary linguistics common to all three languages (e.g. overlapping vocabulary or sentence structuring), but we cannot expect the model to perform well, translating English to French.

Practical Note

You should only compute the transformation statistics (e.g. mean and variance for normalisation) only on training data and use these values to then transform the training data itself, and then the same values to transform the test data.

Including the test dataset in the transform computation will allow information to flow from the test data to the train data and therefore to the model that learns from it, thus allowing the model to cheat (introducing a bias).

Also, it is important not to confuse transformations with augmentations. Some "transformations" might be used to synthetically create more training data, but don't have to be used at test time. For example, in computer vision, deleting regions of an image. Test time augmentation is something you could read about.

Extra discussion

More complicated models (ones with many many more parameters) might be able to perform some kind of interpolation, especially if your dataset if N-dimensions with a large N (e.g. > 10).

This has recently been seen with extremely large models, such as Open AI's GPT-3, which has 175 BILLION parameters, and is therefore even able to perform quite well on completely different tasks, let alone the given problem in the training set range.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much. Well explained $\endgroup$ – Dave May 31 at 18:45
  • 2
    $\begingroup$ -1: "Out-of-range" seems to miss the point. Even if the model generalizes perfectly outside the range of the training data, it will not give the right predictions if the transform is not applied. Omitting the transform produces a systematic error in predictions that is separate from and additional to any generalization error -- e.g., we use a log transform, fit y = 5*log(x) + 2, and then use y = 5*x+2. It amounts to simply making some arbitrary manual change to the trained model. This will throw off predictions whether or not it takes us outside the range where the model is good. $\endgroup$ – nanoman Jun 1 at 7:26
  • 1
    $\begingroup$ @nanoman - thanks for the comment - I have updated my answer to reflect your input! $\endgroup$ – n1k31t4 Jun 1 at 10:53
4
$\begingroup$

There is only one answer to this question, which is no, it is not acceptable. Whatever transformation you apply to the train data (PCA, scaling, encoding, etc.) you have to also apply to the test data.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thank you so much $\endgroup$ – Dave May 31 at 18:45
  • 3
    $\begingroup$ Adding to that - the transformations are really part of the model. The raw test input that goes into (transform + linear model) is no less 'real' than anything. What the intermediate representation after the transform is is just an implementation detail. $\endgroup$ – Sean Owen Jun 1 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.