5
$\begingroup$

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found like this one and this one that suggest that applying the normalization to the test set is not really required and it might depends on many factors such as the model used for training or the nature of the test data.

Now, personally, I am more inclined towards applying the normalization on test data as well. But the problem is this: I am working on a neural network model where:

  • If I apply normalization using the recommended way I get 79% accuracy, (and to be honest it's not interesting for me)
  • If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

So, I don't know what my neural network performs better on test unseen data if I use the second method. I really I want continue using the second method for this particular model, but I don't feel good about it and feels like it's wrong or cheating.

Now, I have one last argument. The last link I provided, have one answer that says this:

"..This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine..."

The dataset I use is a refined version of its predecessor (NSL-KDD dataset). The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

Ps: Sorry if this is long, it's a research ethics thing. I will follow the approach you guys recommend.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
6
$\begingroup$

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a prediction for a single instance independently from any other instances, in which case there is no set of instances to obtain the mean/std dev from. More importantly, the prediction of the model for a given instance should always be the same. Normalizing on the test set breaches this principle, because it makes the prediction for a particular instance depends on the other instances in the test set.

I don't think "separate normalization" is unethical strictly speaking, because it doesn't imply using any of the test data at training stage (whereas normalizing before splitting the train/test sets would). However it's theoretically incorrect for the reasons I mentioned above.

The fact that you obtain such a big difference in performance by normalizing "separately" points to a very different distribution of the data between training and test set (or a bug somewhere along the process). I'd suggest investigating that, maybe there's some error in the data?

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your answer. For your question, I don't think so, the test and train are provided separately, I fused them only to perform some pre-processing steps : Do One-hot encoding on some categorical features. But I did NOT fuse them when I perform normalization. One thing to note is that for some categorical features, some features have values that are present ONLY on test set. So when I perform one-hot encoding on the fused data set and then separate them. The train set will have some 0 rows. So probably the 0s harms training for the first approach ?? I don't know, it just an idea. $\endgroup$
    – U. User
    Sep 5, 2019 at 14:00
  • 1
    $\begingroup$ It's indeed possible that the values which don't exist in the training data cause the issue. But also this points to the more general problem: the training set and test set are not drawn from the same distribution (i.e. they are not a random split of some full original dataset), which means that other features probably also have a different distribution. This is what causes the problem. $\endgroup$
    – Erwan
    Sep 5, 2019 at 14:49
  • $\begingroup$ So what do you think I should do? As I said, the train and test set were originally separated, all I did was perform preprocessing steps and develop my model. Right now, I am investigating the possibility of feature selection on the train set to see if some algorithm can solve the issue of 0 values. What do you think? Any other suggestions? $\endgroup$
    – U. User
    Sep 5, 2019 at 15:31
  • $\begingroup$ It depends to what extent you are tied to using the original test data: an easy fix would be to create your own test set, by splitting either the training set or the merged training+test set. However it might have been designed like this on purpose to represent the challenges of real-life cases... afaik the theoretical framework of supervised ML doesn't really account for that. $\endgroup$
    – Erwan
    Sep 5, 2019 at 16:05
  • $\begingroup$ It was definitely designed this way and a lot of researchers use it without changing the data. However, they don't perform the one hot encoding method to pre process the categorical features, they use instead numerical encoding which gives a smaller matrix with no 0 vectors. It sounds better, but the features are independent and it's not recommended. As I said, I am only trying to follow the ML good practices. I'll stick to the first normalization approach and to the pre processing steps I already use and see if feature reduction can help. Thank you for your time and effort again. $\endgroup$
    – U. User
    Sep 5, 2019 at 16:52
4
$\begingroup$

You need to normalize the test set using the parameters from the training set.
The purpose of model evaluation is to answer the question, "What performance should we expect from this model if it were to be used on a real problem?"

With that in mind, consider how your model would be trained and used in a real application: You would train the model using all of the available data and save the normalization parameters. When a new, unlabeled example comes in, you would use the saved normalization parameters to transform the example, feed it through your model, and generate a result. To simulate this process, you need to normalize the test set with parameters found in the training set. It's the honest and accurate way to evaluate a model.

Regarding the quotation you mentioned:

This is all dependent on size of data sets & whether both train and test are equally representative of the domain you are trying to model. If you have thousands of data points and the test set is fully representative of the training set (hard to prove) then either method will be fine...

A similar quote from the reddit thread:

If your training set and testing set are drawn uniformly from the same distribution (i.i.d.) and your dataset is large then the mean / std on the train and test set should be pretty similar.

These statements are true, but not helpful. If the test set is "fully representative" of the training set, then the training and test sets will have nearly identical mean and standard deviation. So the results of normalization will be virtually identical. I don't think this condition is met in your data set, since you're seeing a large discrepancy in the results of the two normalization approaches.

The authors said "There is no duplicate records in the proposed test sets" and that they have removed any redundant values. So I feel, this dataset is uniform and the test set actually representative according to the authors. So can I use the second approach?

The authors of the dataset have only stated that duplicate values have been removed. They do not claim that the test set is representative of the training set. So I don't think you're safe to use the second approach.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your answer, I will use the first approach after I re-verify the processed I followed to develop the model. $\endgroup$
    – U. User
    Sep 5, 2019 at 14:02
2
$\begingroup$

There are two kinds of normalization.

The usual kind is where you scale some column in the data set, commonly either based on min and max or using the mean and standard deviation. In these cases you really should use the exact same parameters used during training. In fact, you will want to treat your test set as N sets containing 1 object each.

The other kind is frequently seen in image processing, where you would enhance the contrast in an image or even a patch of the images. In such cases where the entire process is local - always using obly one sample at a time, then you only need to employ the same procedure both for training and test.

Make sure you do not leak any information from test to training via normalization. It's a very common mistake to first normalize, then split the data...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.