Why models performs better If normalize test data and train data separately?

Question

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found like this one and this one that suggest that applying the normalization to the test set is not really required and it might depends on many factors such as the model used for training or the nature of the test data.

Now, personally, I am more inclined towards applying the normalization on test data as well. But the problem is this: I am working on a neural network model where:

• If I apply normalization using the recommended way I get 79% accuracy, (and to be honest it's not interesting for me)
• If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

So, I don't know what my neural network performs better on test unseen data if I use the second method. I really I want continue using the second method for this particular model, but I don't feel good about it and feels like it's wrong or cheating.

Now, I have one last argument. The last link I provided, have one answer that says this:

"..This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine..."

The dataset I use is a refined version of its predecessor (NSL-KDD dataset). The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

Ps: Sorry if this is long, it's a research ethics thing. I will follow the approach you guys recommend.

Answer 1

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a prediction for a single instance independently from any other instances, in which case there is no set of instances to obtain the mean/std dev from. More importantly, the prediction of the model for a given instance should always be the same. Normalizing on the test set breaches this principle, because it makes the prediction for a particular instance depends on the other instances in the test set.

I don't think "separate normalization" is unethical strictly speaking, because it doesn't imply using any of the test data at training stage (whereas normalizing before splitting the train/test sets would). However it's theoretically incorrect for the reasons I mentioned above.

The fact that you obtain such a big difference in performance by normalizing "separately" points to a very different distribution of the data between training and test set (or a bug somewhere along the process). I'd suggest investigating that, maybe there's some error in the data?

Answer 2

You need to normalize the test set using the parameters from the training set.
The purpose of model evaluation is to answer the question, "What performance should we expect from this model if it were to be used on a real problem?"

With that in mind, consider how your model would be trained and used in a real application: You would train the model using all of the available data and save the normalization parameters. When a new, unlabeled example comes in, you would use the saved normalization parameters to transform the example, feed it through your model, and generate a result. To simulate this process, you need to normalize the test set with parameters found in the training set. It's the honest and accurate way to evaluate a model.

Regarding the quotation you mentioned:

This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine...

A similar quote from the reddit thread:

If your training set and testing set are drawn uniformly from the same distribution (i.i.d.) and your dataset is large then the mean / std on the train and test set should be pretty similar.

These statements are true, but not helpful. If the test set is "fully representative" of the training set, then the training and test sets will have nearly identical mean and standard deviation. So the results of normalization will be virtually identical. I don't think this condition is met in your data set, since you're seeing a large discrepancy in the results of the two normalization approaches.

The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

The authors of the dataset have only stated that duplicate values have been removed. They do not claim that the test set is representative of the training set. So I don't think you're safe to use the second approach.

Answer 3

There are two kinds of normalization.

The usual kind is where you scale some column in the data set, commonly either based on min and max or using the mean and standard deviation. In these cases you really should use the exact same parameters used during training. In fact, you will want to treat your test set as N sets containing 1 object each.

The other kind is frequently seen in image processing, where you would enhance the contrast in an image or even a patch of the images. In such cases where the entire process is local - always using obly one sample at a time, then you only need to employ the same procedure both for training and test.

Make sure you do not leak any information from test to training via normalization. It's a very common mistake to first normalize, then split the data...