3
$\begingroup$

I have seen many people handle the missing or inconsistent data in both their test and train data sets. Sometimes they handle only the train data sets and sometimes they merge the train and test data sets and handle the missing parameter.

So, what is the best approach and What's the difference of these two? Does any approach effect the predictive model or it's just optional or good practice to handle missing data in both data sets.

$\endgroup$

1 Answer 1

2
$\begingroup$

There are a few things that you need to be careful with here.

You can do certain things when preprocessing data or performing data augmentation that can be applied across an entire dataset (train and validation). The main idea is not to allow the model to gain insight from the test data.


Time-series example

Missing data can be managed in many ways, such as simple imputation (filling the gaps). This is very common in time-series data. In your training data, you can fill the gaps using the previous value, the following value, the average of the data or something like the moving average. Where you must be careful is with violating the information flow through time. For example, in your test data, you should not fill gaps using a method that looks at data points in front of the emtpy time slot. This is because, at that point in time, you will not be able to do the same as you shouldn't know the future values.

Image data example

Looking instead at image data, there are data-preprocessing steps such as normalisation. This means just scaling the image pixel values to a range like $[-1, 1]$. To do this, you must compute the population mean and variance, which you then use to perform the scaling. When computing these two statistics, it is important not to include the test data. The reason is that you would be leaking information the dataset that is then used to train a model. Your technically knows things that it shouldn't; in this case, clues regarding the mean and variance of the target distribution.


People might also consider "missing data" to include imbalanced datasets; i.e. there are cases that you know of, but just don't appear in your dataset very often. There are some tricks to help with this, such as stratified sampling or cross-validation. The optimal solution would, of course, be to gather a dataset that more closely represents the problem at hand.

$\endgroup$
2
  • $\begingroup$ Hi. So it's important not to include test set during processing? But for image data, if I use some basic image processing using OpenCV for both training set and test, is it OK? Processing like using few Gaussian filter or OTSU method or dilate etc...Well, doing this only on the training set produce a great result on both training set and validation set. But the terrible result on the test set. $\endgroup$
    – Innat
    Mar 5, 2019 at 8:45
  • $\begingroup$ @iPython - any processing you can actually do on inseen images (during inference) is ok to do on the test validation set for training. It is really popoulation statistics you use that should not contain any information from the validation/test sets. $\endgroup$
    – n1k31t4
    Mar 5, 2019 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.