My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test set is 'locked away' during training time. Should I peek at the test data to see if there are missing values, or should I fit imputation for all features on a complete training set?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
First, we must understand about a common statistical term called population. Given a population say X, a random sample is drawn (in the ideal conditions). Now suppose you are asked to build a predictive model based on this random sample. So, you split the sample into train, test and validation sets. And you start to build the model on the train set. You begin with the initial data cleaning activities, and you find there is no missing data in it.
With this brief background, now to answer your Q literally forces me to rather ask you a question;
Q. Did you collect the data or was it given to you?
If your response to the former part of my Q is yes then by all means do whatever you want with the data. And if your response to the latter part of my Q is yes then you should not make any attempt to pollute it. Nor should you make any attempt to peek into the holdout dataset, because then subconsciously the predictive model you have built will be sabotaged.
Finally, remember the concept of population and sample. It holds the key. Build your model such that it accounts for all initial data preprocessing activities. If a particular step does not fulfill the dataset, then code can be written to avoid it
Edit 1
Basis of the OP suggesting the dataset was given to them, I'm revising the answer further;
i think your getting confused between the population data and new data. As per my answer, models are built basis of the population data. The new data should have the same attributes and properties as the population data. However, if for some reason unknown to you, the new data has missing values but the original data does not, then there are a couple of options, namely;
a. Determine the missing data pattern,
i. if its missing completely at random (MCAR), means there is no relationship between the missingness of the data and any values, observed or missing data. In other words, no systematic differences exist between participants with missing data and those with complete data. In these instances, the missing data reduce the analyzable population of the study and consequently, the statistical power, but do not introduce bias: when data are MCAR, the data which remain can be considered a simple random sample of the full data set of interest.
ii. Missing at random (MAR). When data are MAR, the fact that the data are missing is systematically related to the observed but not the unobserved data.
iii. Missing not at random (MNAR). When data are MNAR, the fact that the data are missing is systematically related to the unobserved data, that is, the missingness is related to events or factors which are not measured by the researcher.
Once you have determined the nature of missingness, then you should determine if the missing data can be imputed or not.
In conclusion, if the new data has missing values then you can treat them or else remove them. The choice is yours. My suggestion will be to follow the middle path, in which you can build 2 models; first model without the missing data and second model with the imputed missing data. Evaluate and validate both the model performance and choose the one which closely matches the desired business/research outcome.
-
$\begingroup$ So suppose all data we have is given to us. Then your recommendation is to avoid 'pollution' (which I assume means a implementing a missingness measure like imputation), and also avoid peeking at holdout data. Then if, at test time, we are presented with missing data and we are unable make a prediction, we would retroactively implement a missingness measure on the training set to allow for this prediction? Is my understanding accurate? $\endgroup$ Commented Sep 10, 2020 at 6:53
-
$\begingroup$ @Omoplata7C0 check my answer, I have revised it. $\endgroup$– mnmCommented Sep 10, 2020 at 7:04
-
$\begingroup$ Thanks for the revision. If we build 2 models, one with imputation and one with omission of the missing data, we choose the model based on test set performance? Because the score for both these models will be identical on the training set (since there are no missing values), we will be choosing the model based on the test set score. If the missingness pattern is MCAR or MAR, this is fine to do, as this will not introduce bias, correct? $\endgroup$ Commented Sep 10, 2020 at 8:38
-
$\begingroup$ Also to clarify, my training set and test set are fixed. I know for a fact that the training set contains no missing values, but the missingness of the test set is unknown, as I have not seen it. My plan is to perform nested cross validation on the training set for model evaluation, with the final model being built using all training data. The final step is merely evaluating on the test data. With your recommendation however, it seems I will be using the test set to choose the model. $\endgroup$ Commented Sep 10, 2020 at 8:42
-
$\begingroup$ @Omoplata7C0 you choose the model which answers the Question that is asked. And you must inform the project commissioner on the reason you chose the particular model. And I still fail to understand why must the test data be different from the model you are building. And if I were you, I would question the project commissioner on this particular anomaly and its underlying reason before building any models. $\endgroup$– mnmCommented Sep 10, 2020 at 9:26
$\begingroup$
$\endgroup$
If the Train data(~80%) doesn't have any missing records and you are expecting missing records in test data(~20%).
This can happen in these circumstances(can be other too) -
Only few missing records in the count -
Then these are most probably completely at random, then you can either remove the records or fill with the mean/median of training data
A Good number of missing records in the count -
It means the Training set is not representing the dataset properly. Then anyway your model will struggle with this Test data and you might have to create a better Train/Test set.
Also, chances are high that missingness will fall under the other two Categories. So you may impute accordingly.
