45
$\begingroup$

I am doing a project on an author identification problem. I applied the tf-idf normalization to train data and then trained an SVM on that data.

Now when using the classifier, should I normalize test data as well. I feel that the basic aim of normalization is to make the learning algorithm give more weight to more important features while learning. So once it has been trained, it already knows which features are important and which are not. So is there any need to apply normalization to test data as well?

I am new to this field. So please ignore if the question appears silly?

Improve this question
$\endgroup$
1
  • 8
    $\begingroup$ Your test data should be in the same scale as your training data. $\endgroup$
    – Jon
    Commented Feb 8, 2018 at 16:58

2 Answers 2

Reset to default
72
$\begingroup$

Yes you need to apply normalisation to test data, if your algorithm works with or needs normalised training data*.

That is because your model works on the representation given by its input vectors. The scale of those numbers is part of the representation. This is a bit like converting between feet and metres . . . a model or formula would work with just one type of unit normally.

Not only do you need normalisation, but you should apply the exact same scaling as for your training data. That means storing the scale and offset used with your training data, and using that again. A common beginner mistake is to separately normalise your train and test data.

In Python and SKLearn, you might normalise your input/X values using the Standard Scaler like this:

scaler = StandardScaler()
train_X = scaler.fit_transform( train_X )
test_X = scaler.transform( test_X )

Note how the conversion of train_X using a function which fits (figures out the params) then normalises. Whilst the test_X conversion just transforms, using the same params that it learned from the train data.

The tf-idf normalisation you are applying should work similarly, as it learns some parameters from the data set as a whole (frequency of words in all documents), as well as using ratios found in each document.

* Some algorithms (such as those based on decision trees) do not need normalised inputs, and can cope with features that have different inherent scales.

Improve this answer
$\endgroup$
7
  • 3
    $\begingroup$ +1 for explaining that normalization parameters for the test should be the same as those determined from the training. It’s my understanding that this logic extends to any validation sets as well. $\endgroup$
    – Greenstick
    Commented Feb 9, 2018 at 19:30
  • 1
    $\begingroup$ @Greenstick: Yes of course, it extends to treatment of all data fed into the model. Including new data if a model will be used for inference in production. I think the answer is long enough without covering all that though, and the point you split off validation data varies a lot (many models have this built in to training step), so could not show code example. $\endgroup$
    – Neil Slater
    Commented Feb 9, 2018 at 19:42
  • 4
    $\begingroup$ Should you scale(fit_transform) the test data WITH the training data? Or do it separately using the same scale(scaler.transform)? $\endgroup$
    – Bob Ebert
    Commented Feb 9, 2019 at 4:50
  • 6
    $\begingroup$ @BobEbert: You can do either. Technically you may be leaking a small amount of information from test to train, if you do fit a scaler to the combined data set - so the "official" version could be to fit the scaler to training data only, and apply it to all other data sets thereafter. However, this leak is very minor and I have never seen it cause a problem in practice if you fit the scaler to all data you have. $\endgroup$
    – Neil Slater
    Commented Apr 15, 2019 at 20:21
  • 1
    $\begingroup$ @HammanSamuel Yes it does, for same reason. When you train on normalised data, you set the "units" that your trained function works with. All input to it should be in the same units so that the function works as expected. $\endgroup$
    – Neil Slater
    Commented Apr 12, 2020 at 6:42
13
$\begingroup$

Definitely you should normalize your data. You normalize the data for the following aims:

  • For having different features in same scale, which is for accelerating learning process.

  • For caring different features fairly without caring the scale.

After training, your learning algorithm has learnt to deal with the data in scaled form, so you have to normalize your test data with the normalizing parameters used for training data.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.